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ABSTRACT 


These proceedings report the results of a workshop on space telerobotics , 
which was held at the Jet Propulsion Laboratory, January 20-22, 1987. 

Sponsored by the NASA Office of Aeronautics and Space Technology (OAST), the 
workshop reflected NASA's interest In developing new telerobotics technology 
for automating the space systems planned for the 1990s and beyond. The 
workshop provided a window into NASA telerobotics research, allowing leading 
researchers in telerobotics to exchange ideas on manipulation, control, system 
architectures, artificial intelligence, and machine sensing. One of the 
objectives was to identify important unsolved problems of current interest. 

The workshop consisted of surveys, tutorials, and contributed papers of both 
theoretical and pratical interest. Several sessions were held with the themes 
of sensing and perception, control execution, operator interface, planning and 
reasoning, and system architecture. Discussion periods were also held in each 
of these major topics. 
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PLENARY SESSION 


Program Objectives and Technology Outreach 

P.S. Schenkcr 
Jet Propulsion Laboratory 
California Institute of Technology 
Pasadena, CA 91 109 



NDrtshop p r ocee di n gs 


-f'V" « 
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in telerobot ics 
the content and objectives of the 


VTlrn nf the nrkahnp are — iy i — these topics range from b^lc telerobotics renenrch isenea to design 
and integration of a near-term flight initiais«cferational capability s pac e telerobot. Among the areas to be 
covered are: machine vision and tactile sensing. aanlpulator control, Al-based t ask planning and error 
diagnosis, teleoperation and human-robot interfaceXand teJerobqt system architecture and design con c ept s . The 
central t ech nology t h eme is the supervised autaaaticrKqf spao^/ 'operations, with eschasis an scenarios involving 
platform or spacecraft servicing, assembly, and repair. 


The objectives of the workshop are threefold: 
telerobotics research we are doing and the prograraatic, 
-*• hope through the technical sessions to stimulate 
telerobot design and i^&lemsitatlon . Finally, and 
and perhaps yet wmddreassd prnhlt— confronting 
Thweed the last wh jsst lim. the imekahsp will 
panelists offei^m~bH5pective on tmlerobqtics 
vj^jpciint s i&fti tftte a ctivitie s^ 


would like to convey the spirit of the 
Irectloi^iyMe expect it to take in the future. Seoul, o/ 
ically fecu frag discussions an contrasting a pp r oa ch es to 
Identify some of the important unsolved 
development of ^viable s cace telerobotic capability. 


r an UKrrraa - 
requiring 


i^nil of’ ^stin^uatad- 


NASA Activities in Telerobotics 

NASA at this tiae has three principal areas of^detivity in telerobotics. The 'first is the Office of 
Aeronautics and Space Technology (OAST A Code R) Tplerobotics Research Prograa. The second activity is the 
Office of Space Station (OSS - Code SJ Flight Tfel^robot Servicer Engineering Oevelooeent^Prograa. The third 
activity, s up p or te d by eeny NASA Offices and Cmmrs, is various problem-specific teierobot\teign studies and 
technology assessaents. This last category ipdludes applications such as the Shuttle, pla n eta r y rowers, orbital 
aaneuver vehicles. The o— rn goals of lam se activities is the tiae-staged development of ^increased space 
mission productivity, ex pa nded mission capability, and ultimately extende d mission autonomy. Enhanced safety of 
sp a ce operations is a unifying jam mm j£ramm these goals. The planned discussions and presen tatjens of this 
w orksh op largely focus cn activity of/the Code R NASA Itelerotootlcs Program. This pnjgrm m has Its 1965 origin in 
a set of derived system requirement^ smmr lari in Figure 1. In ceroe n ce the pr ogr a m is a "technology push" 
effort: believing that technology/exlats or can be defined in the near-to-mid term to impact the above stated 
mission goals, NASA Code R is investing in dawloping that technology to ground laboratory breadboard proof-of- 
ooncept. {By msaranda of agreement between Codes R and S. elements of this technology are to be transfe r re d to 
the Flight Telerobot Servicer P roga m for s ubs eq u ent evaluation and advanced development.) 


•Dr. Sc h enk c r is Chief Technologist of the NASA/JPL Teierobotics Project. His mailing address is Jet Propulsion 
Laboratory, 4800 Oak Grove Drive. M/S 198-330. Pasadena. CA 91109. 
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Figure 1. Telerobot Demonstrator Requirements Development 





The taak dr— In of the space ttlcrobot fa— a nab«r of defining d — c tg te Uci; 


at Maekpiac— : 

b) 111— Inatlon; 

c) Obs er v ati o n : 

d) Oraplaarlty: 

•) ODoartainty: 
f) Manipulation: 

9 ) Mobility: 

h> Cr— ilratimM 


Highly variabla 

Indirect. vldao or tala present 


Structur e d to highly u ns tru ct ur ed 
m— Inq data, —11— IDoaly 


Ex te nde d — I/or f laadfala stractun— 

My be required for tr a waia a car foil w o rks pa ce 
Possible out-of loop delay re: run-ti— control 


Relative to the— characteristics, ms preview two a wr re— a telerobot ayat— s cen a rios: 

(1) Initial Operational Capability (IOC) : * Stationary two-arm telerobot p a rf o— a known staple tas ks on 

cooperati— sp a cecr aft using h and and power tools ... limited autooosy. 

(a) Stationary telerobot, stabilized task fr— e, find w ur k sp a ue 

(0) Stereo pr es ence and force feedback 

(c) Detailed CM) w orksp a ce ref e i cnc e 

(d) Controlled and predictable 111— 1 nation 

(e) Aute— ted workpiece acquisition and verification 

(f) Coordinated handling of ext e nded objects ttoich have well-defined grasp points 

(g) Operator dswi*yniteri or control -dr i^n autonstlcn/ teleope ra tion trad e s 

(h) Sequential, highly structur e d task schedule 

(2) Technology Outreach: "... Cooperating aobile telerobots p e ifQL S co^tlex teepoorary and per— nt repairs 
of aged el— Bits using auxiliary supports, guides, and power tools... Periods of autonoey m easur ed 
in me— nts...” 

(a) F r ee flying robots, aoving task frases. variable workspace 

(b) Natural language di sco urse, single ope ra t or supervision 

(c) Mixed aabient ill— inaticn and active sensing 

(d) Automated acquisition, tracking, and 3-D interpret a tion of w or kspac e 

(e) Inter-robot, coopers tive handling of generic objects 

(f ) Cooperative wor ksp ace tra— — required 

(g) Auto— tad load-aharing/sheddlng betw e en operator and robot 

(h) Parallel, dynaaically allocated task schedule 

(1) Event-driven, auto— ted replanning of task in rw-tise 


The IOC scenario suggests the laboratory floor telerobot capability that we can end should try to establish 
as a technology baseline. The NASA Code R p ro gr— has planned such an IOC d— on atia ticn tdiich will be discussed 
in detail later in the p ape r . The second scenario of "technology outreach” has very advanced technology 
features, sc— or all of tkiich one might e x pe ct to characterize a hiqkily evolved telerobot capability. Such an 
evolutionary telerobot would far outstrip the IOC syst— in task scope, capability, duration of autnnr— 
operation, and richness of -aachine interaction. The primary r ea son for introducing the— scenarios at this 
time is to motivate understanding of how and ttiy the XASA Code R Telerobotics Pr ugi— has been farsolated, which 
is the next topic. 


NASA Telerobotics Dwen rr*i Pr ogr— (Code R) 

For many years NASA has conducted and s u ppor ted res ear ch efforts in robotics relatsd are— such — 
manipulator design and co n trol, teleroperations . human f a ctors and hi— n — rtilnr interface, and AI planning. 
Concern tan t with initiation of the Space Station Progr— and related n i i— i nriitions of the Advanced Technology 
Advisory Ccnmittee, NASA OAST sought to unify these re se arch efforts into a focused Auto— tion and Robotics 
technology devel o p— i t. TO this end, two progr— s — initiated, Syste— Ante— tion and Telerobotics. The 
NASA lead centers far the— pr gr— s are r espe ct ively Aaes Research Center and Jet Propulsion Laboratory. The 
two piugr—g, have a co —c n ba— in five sustaining core research thrusts and the goal of providing ground 
laboratory aeaonstrations of system- level breadboard technology readiness . Both pr ogr ams are under the NASA 

headquarters s— ervisicn of lee B. Holcoab. Director of Code RC, I nfor mation Selene— and fl — n Factors. 

The major elements of the NASA Telerobotics Research Progr— are — folio—: 

a) A fiv e y e a r RfiD plan derived by a NASA-wide Telerobotics working Groip 

b) A cooper a tive NASA center Ami versi ty/ industrial effort 

c) A syst— integration t es tbed aaintainsd at JPL 

d) Planned periodic demonstrations of an eiroiving syst— capability 

e) NASA technology tr an s fer via the Flight Telerobot Servicer 

f » Periodic p rogr — review by an exter na l Telerobotics Technology Advisor Co— it tee. 
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In addition to core mwch activities, ditch will bs dear ri hart In g re ater datail *hnrtly, tbs Cods 1 
lalerabotlcs Program ftnb tbs development of a ay— a integration ta s tbad. Tbs wain tbaaa of tbs testb e d la tbat 
tbs definition and intmeticn of system level control for a telerobot is a vary challenging r aaa arch prabla in 
itself: to meaningful!/ address this prnhla, both in co nce pt fomnlation and evaluation, we believe such a raal 

datlof t anelraoeant ia aasm tlal. 


The five MSI coca Antcmat Ion and Robotics (MR) reaesrch th r usts ara — arlaed in flgot 2. Tbaaa fhe 
research thrusts, as interpreted for the TSlerobotics H i ij i ■ ere i n tende d to ptwida the tech n o logy baas free 
tdiicb to integrate an evolving telerobot taatb a rt . Figure 3 at— r 1 css the participation of the various MSA 
ce nte r s in the telerobotics care raaaarch; Figure 3 la illustrative cnly - it is neither coapratsnslve nor does 
it address r aaa arch specific to the Systems Jtatnaation prog—. 


S e n si n g and pa r c apt icn deals with acquiring a perception of the anvlranmnt. specifically determining the. 
geometry, dynaaics, idnxtity and relaticnshipa of ob j e cts . The apbaait is dynmalc work piec e acquisition and 
stipulation, i.e. , to acquire, track, physically stabilise, and verify obje c t s of Interest. 


Task planning and reasoning is the arse of logical planning and spatial reasoning of activity eaqunoaa 
together with a strong concern for the rod— s of psrfm— a and the efficient use of ta— n naddna 
raaouresa. A rather challenging problem in this repud cen c s roa i nt e g r at ing sequential (or even parallel) 
activity planning with spatial reasoning and coordination. An o th er imp ortan t problem involves integrating into 
the overall system architecture an eisuitlm sequence aonitoring strategy tnich goes beyond si^le e rr o r 
diagnostics. Error cor rection say require physical reasoning and b a ck tra ckin g of er ror in a dynamic replan sods. 


The area of operator interface has to do with the interaction between the tammn st*xrvlsar end the 
autonomous entity, the robot. The —hauls la on cognitive and physical work load reduction, end deals with not 
only teleoperetad or r obotic modes, but also abased task sodas. Task state p r esentation is thematic in this 
development. Dext erou s hand co n trol la a aajor p rog — focus; telepreeent feedback through st e re o vision force, 
tactility and various aspects of cross aodal end virtual display ara also active areas of is ss arch and 
development. 


Control execution ant-passes all s ens or ref ere n ced aliments of both the Manipulator and platform control. 
Control schemes here say include not only the classics! position and force/torque control, but also acre 
sophisticated adaptive and actively ccapliant control. There ia a strong focus an the synergistic develofmmxt of 
advanced con tr ol laws end their real-ties iapl— itaticn. This effort includes integrated amort and e f fec to rs 
end various a sp ects of high-speed, six d eg r e e of frwiVs teleoperatlcne. We are pursuing elements of both hybrid 
and adaptive control, including a mmg eaent of manipulator redundant degre es of freedom, and a Immure of multiple 
eani pula tor equal-status c o n trol. 


The fifth core area, system architecture end integration addressee: a) system-scale fractional allocation 
end coordination of siTimarml task sequraoee, b) rra-tise exchanges bet w e e n automated and taiaoparatitm control 
modes, c) more ad v ance d c once pts for afaar ad functions between operator and robot, and d) hi erar ch i ca l and 
distributed computer i^lcsentaticns of the abeam. 


Teierooot Testbed 

As the coordinating center far the XASA Code R Tblerobotics Program, JPL is developing a boat-site test - bed. 
The objectives, ap pr o ach and significance of the system integration testbed are aa follow: 

(1) OBJECTIVE: Define and develop telerobot system-level arcltecture for variably structured apace 

servicing, irsmhiy. repair operations 

(a) End-to-end hierarchically i nteg r a ted rra-tise co ntro l 
<b) Run- time exchange of autcoatira/teleoperatiane 

(c) Knowledge base generation and mitresfart activity planning far ras tructured wo rksp a ce 

(d) Dynamic sen s or allocation strategy far management of rarteep a o e u n cer tainty, execution errors, end 
anomalous conditions 
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Figure 2. NASA Core Reeearch Supporting Telerobot Development 






(2) MIOKS: Vmmm x hraq* csniar-cf awnellanre based on lmt-aite eyataa letagrf km tmt but 

B contracts 

(d) teeth ed dmlqpt plan for nobriag n»f capability 

(•I Coda MbdH S technology tr an sfe r pin to guide at lam p wd a ct mmoe 
(f) Inrfcat rlal cuuti a c ta to Aiw ap t — aptmtag ^ p wdi 


(3) SZGKZFZCANCS: 9 ys t ame t i c MQlntiCB 



and wrtfkaUia of a telerobot technology coooapt tiktdi dlractly 
domain. and opantlaal laitahl llry larpil ream its of 


(a) Fully idntifi« the r a e aa rrh need e d to togat ooaponant/cocw technologi es 

(b) Stimulates contributions and inolmat of p e a r re se arch ooaounity 
(c| Mm eaplicit contribution to tarsmtrlal MR o ff or t 

(d) Provides technology fsartthrmofr Mm Rmr Wharton 

(a) Insures that OdST MR Research effort dlractly addresses rmjnli genre and d e si g n iaaoaa of 
flidR/dpUatlon ca n t a r a 


The ala of the t o othe d la to define an evolving talarabot e y e tea level anc-u* jcture for apace servicing, 
aaeaably and repair ope rati ons. Enabling on a pranh for thla ca pab il ity Include, for m a ple , a hlarsr rhl cal 
Integrated run-time control architecture, and an interactively entreated generation of knowl e d g e faeee and 
activity plane for taeklng In an ratructurad workspace. da s t a te d earlier, the ap proa ch usee NASA cen ters of 
aacsllsnrs for penetrating selected research areae thet s u ppor t the I n tegration teethed. Ih eea technologie s will 
he developed to the breecRjoard level. An laportent el amt here la the collaboration of NASA cantara with 
i n d ustry end uniformity contractors. 


The planned capability of the 1967-1990 telerobot teethed is a mar lie i in figure 4 end e la bor at ed in 
figures 5 and 6. In 1967. ms plan the development of a tank automation capability in which trajectory control 
and elaaanmry off-line task planning are i nteg rated and aerified for beelc eae cutlon aa ryarr ee on a elw>le 
taakhoard. In 1966. a full 6-DOF bilateral force-reflecting te leop ara ticna capability will be integrated with 
an lypaiei varaion of the autoeetim aub-architactuze. This initial capability talarabot will be eaerciesd in a 
satellite repair servicing daamatratfm derived free the Solar Mariana Hi salon repair scenario - aae flguraa 7 
end 8. Work leading to the 1967 am treat Inn capability la largely a JPL in hnea effort. The 1986 telerobot 
denial, lap Imitation and integration work is collaborative with ekenal eye tear contractors. In 1969. we 
ep e.t to substantially incxaeee narfremanra of the 1966 telerobot t hr oug h fractional anga»nfat1rms in noet 
telerobot eubeyeten arses. This effort will be leve ra ged by initial technology deliver las firm other 
participating NASA cantara end free JPL co n tre e to re. Deliveries planned to have a major functional impact 

include strategics for cooper a tive ( equal -status ) dual am aanlimlafnr control, active ooapiiance aanipulation 
strategies, AI -planning capability allowing aom d e gr ee cf on-line error racowery. and an ea cp e nri s fl machine vision 
database. In 1990. the testbed is planned for developeeuc to its second eajrr daaonetmtion. This evolution 
demonstration la planned to i n corp ora te mobility of both the taekfaneid end moot servicing platform. 


Tb give the reader a better sense of the d es ign ap p ro a ch to the testbed, we now present a brief overview of 
the teethed architecture. These following remarks are directed to the 1968 telerobot design goal taut are 
c on cep t u ally valid for the overall 1967-1990 testbed developaent mercies. The reader any find it helpful to 
refer to Figure 6 during the various s u bs ys t e m disci, mime: 


New we turn to a acre detailed view of the 1966 demonstration and control struc t ur e (see Figure 7). The 
abjective in 1966 is the integration of the telerobot. Figure 6 depicts the teak b o a rd for the deems t ration . 
It is a simulation of the Solar Hex satellite. The satellite rotate s slowiy . The full u nla b el l ad object is 
mooed, acquired, and staoilxaed. Then, as necessary, it is rotated for wo r kplace verification end a oyrtsi ticn. 
Dual are coordinated control is required for leplssenting aany of the teaks defined for the ilmiwf retlm. Also, 
tne capability for a nn-tiae traded autenonous/ teleoparative control will be danue listed. 
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Figure 4. Telerobot Capability Integration Plan 





Figure 5. System Integration Testbed 























Figure 6. Functional Architecture 



STATIONARY ROBOT, SIMPLE SPACECRAFT 
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Figure 7. 1968 Telerobot Demonstration 
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MODULE CHANGEOUT PROTOTASK SEQUENCE 

• ACQUIRE. TRACK. AMD STABILIZE TASK COORDINATE FRAME 

• OPEN THERMAL BLANKET, SECURE 

• ACQUIRE POWER DRIVER 

• LOOSEN CAPTIVE SCREWS WITH POWER DRIVER WHILE HOLDING COVER 

• STOW POWER DRIVER 

• REMOVE AND STOW COVER 

• RELEASE MOOULE RETENTION CLAMPS 

• WITHDRAW AND STOW MOOULE 

• ACQUIRE ANO INSERT REPLACEMENT MOOULE 

• TIGHTEN MOOULE RETENTION CLAMPS 

• ACQUIRE COVER 

• ALIGN AND HOLD COVER 

• ACQUIRE POWER DRIVER 

• TIGHTEN CAPTIVE SCREWS WITH POWER DRIVER 

• TEST SCREWS 

• STOW POWER DRIVER 

• RELEASE. GRASP THERMAL BLANKET 

• DRAPE BLANKET 

• FASTEN BLANKET 

• INSPECT 


Figure 8. 1988 Telerobot Denonstration Taskboard 
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I to talenobot allow tbt w pt rvi iiflg oper at o r two basic madam of control: tt lK cwitl op nd r obo t 
-ifrwriin Id a th e e , the o pt tcr bee tte ability to aaoothly witch bet— i thaw ndn daring r ra - tlnu took 
wcation. The noawl tok wcitlnn aoado is: 



Should an o ff - a o twl o wn occur daring tank ancotion, ths abos sc— ri o is modified. V^rpnatiot any 
now bs inooiond on an nd hoc basis at any point in tha task a ac aticn tins lint, lbs nost coason r nna rti for the 
oper a tor to dm Igatt talecp a mt lw in— tt loo is d 1 anlopa—t of unaapactad conditions in tbs wnt kapaoa. Ctae 
WMptle is Ilian— y that tbs raakhTsrrt structure dnviataa trcm its GK> rnfnrwca in an ^anticipated way (and 
ca nno t bs nphtad ran tins sensing). Jno tb n r aims’ll a is wcacrectahle error in tbs su—a tsd tank 

sagnanrlng, a.g. . a tool is grossly sisal 1yd or dropped. If tbs ope rator does i nte rr u pt —tad tank 
sequencing with talscparstlcn, bs say yet still Ippl— it subsequent p in— d eutoastlcn sequences without off- 
line rainitial Irstion of tha entire task. This is ml lowed by tha Monitor point” fe a t u re of the tslcrobot 
control architecture. Monitor points are in — mra nominal telarofaot/w oc tap a c a configurations know to occur at 
each preplanned trade of te le n p er aticn-t o m ut—ti cn control . If the o per ator aaauraa that a acnitnr point task 

configuration baa been obtained at the conclusion of a teleoperatiao i nt e rrupt , then he can Monthly release 
control to aufrwtlm In rra tins. 


(Operator control atation - OCS) 

lbs OCS is the physical and cognitive interface with tbs o per a tor . In a reel flight systan scsnrlo it 
r e pr e sen ts tbs "local* or base site fear naira operation of the robot. T be OCS contains the band controller 
i nterfac es for aanaU poeltlcn/rate telecpermticn of tbs robot aanifxumtora. Tbs OCS baa voice-intermctirje and 
ke y bo a r d entry i nterfac es far coawnd/veriflcation of autcatad aaqumcing of the robot aanipolators. Finally, 
tha OCS baa nwrrw textual and grrohlc displays ( Including high-fidelity wide- field s cnoc hrc— tic stereo) Wdch 
provide the operator with both physical and —antic representations of the taak state. The displays can be used 
both for nn tine task sraervisicn and predictive siaolation. 


The IS la a distributed be rt ha ge /software eubeystea vtiich pea f caw three pri nc ipel fractions, first, 
through a centralised systw executive, the IS eatabliafaaa the systw electronic ccnfiguraticn and Instantiates 
the initial task database. Second, the IS enables s ys t w w ide controi/oararaicatlcn via a wiring harness, 
hardkwa/softwra s tk ae yatt a driver interfaces, and local area net (UP) protocol. Finally, the systw executive 
performs systw self-test and calibration at a tart-up, as well as system health aonitoring and data 
col lection/aoni taring daring rra tine. 


(AJ pi— er - AIP) 

The AIP, with ope imtor I nt eracticn. develops the taak script, timeline, and oorad global varlahlae of the 
rra time execution sequence. In the initial deaonstratar operational capability, the AIP p er fuu.ee off-line 
planning and on-line task montoring with a limited capability for hl<£k-lsvel e r ror dlmywsls. Ont year 
deveXopaent for the 1990 evolution daaonatration t ar ge ts and AZP with q— bUity for on-line plaming mad dynamic 
error r — ary . The AIP p er f o r w both logical and spatial planing fractions. The resulting plan specification 
is in the fow of high-level task move sequences: a.g., move right manipulator arm from taak fra— e ffector 
coordinate A to coordinate B through p erm i s si b le taak space fr eewa y volwe V. As part of the plan g— ration 
p roc ess , the AIP identifies the abov e described "monitor points" which are teak configuration* ttfilch guarantee 
safe re-entry from tel ecp e rm ticn to autoaeted sequencing. 


(Rra tine controller - KTC) 

The BTC accepts from the AIP the nominal task plan and sequentially interprets it as « set of r® time 
executable "actions." These actions any be of simple or complex farm, the latter being t er med a "skill." A 
simple action will typically r esul t In a single BTC cowand to the aaniculator control or s e n s ing subs yste m; a 
carol ex action will be i nterpre ted nm an integrated sequence of c— ds to the manipulator control and/ or 
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sensing s ub s yste ms. lbs asln fmcticn of tbs RIC Is oa yutat ion of detailed physical pirns for manipulator 
motion control. These plans are coamuted In ns 1 tins, based on tbs aost currently available task wcrld/stata 
knowledge, and utilize exact geometric models of tbs wor k s pace objects ss well as actual gees e t rlc- kl ne aa tir - 
dynsmical con s traints of tbs robot ssnipulators. Thus, Msen the AIP oowends a nominal effector trajec to ry " A- 
> B thr o m fr freewy V ", tbs RTC nominally does tbs following: 

a) Indexes tbs a ppro pri ate ski 11 /action for the AIP ocamand 

b> Binds the AIP c c am and psrmaetsrs (move sequenoe/gsomstry) 

c) Determines an a p p ropriate terminal stanc e/pose of the sanipulatar(s)/effsctors(s) for the t a s k -specific 

d) Coyotes a collisio n - fr ee arm trm jectory(s) ttar o ugfa the joint configuration space 

e) establishes a joint configuration space waypoint sequence for su b se qu ent i:rsmni cation to the 
silpulator(s) controller 

f ) Parameterizes the man! pula tar (s) controller coamends with appropriate controller node indices and 
limits 


In addition to action-skill cxasmnds, the RIC supports "overrides" and "req u e s t s ". Overrides are softwme- 
driuen execution stops Much occur either at a designated monitor point, or as an internet of a u t o nomous control 
into a halt state from execution of a pre-planned autosation se q uence . The request cca man d does not affect nm 
time motion control; rather, it either a c cesses the RIC data base with a query, or, cammnds c c ap e t s OO P 
supporting a predictive simulation. 


< Manipulation and Control Mechanization - MUM) 

The MbCM subsystem is the lowest layer of the t hree level automation control hierar ch y cc^rised by the AIP* 
RXC-M&CM interface. The RIC oasses to the M0f the above described joint configuration waypoint list and central 
mode/ parameter coamands. The MSO! s ubsequently cooputes a smooth trajectory apprcscinatlon through the waypoints 
and implements this solution via an em b edd e d real-time controller. MfiCM capability is in s om e instances cspec t ec 
to provide simple nuoan-equi valent skills, e.g., grasp and handling of geometrically regular and e xt maw 
objects. To this end. the initial MKX arcnitecture is being developed far a constra ined fora of dual-mn 
coop er ative control: The left arm is rn—rrtad in position control mode; the right arm is rrmmnnrinrt in hytric 
(force/ position) control. Mien a dynamical chain is closed between the t*o arms by their grasp of a ccwor 
structure, the right arm reacts tnrough force/toroue s en s or and position encode r feedback to maintain noaunaJ 
force limits. The underlying accroach to a single-arm hybrid control is derived from the Raibert-Craig approach: 
extension and application of this accroac h to multi-arm co op era ting robots is described elsewhere in these 
Proceedings. 


( Telecp e rations - TOP) 

The TOP subsystem is a distributed computing architecture providing the sensing, control, and disclay 
functions necesary to provide a high-fidelity, dual-arm, six degr ee of- free d om (DCF) teleope r ator capability. 
The gen era l TCP design approach is a bilateral force- reflecting hand controller in Milch the saster and slam 
arms can be both kinematically and dynamically dissimilar. The approach thus allow the independent design of t 
master (hand controller) Much meets the requirements of the operator and local control station, and, slave 
(robot arm) which meets the ne e ds of the remote acoiicaticn environment . The TCP subsystem initial operational 
capability has all DCF controlled by the operator; however, the general cowuter-based ap pr oach allow growtn tc 
shared modes of control in Much sene DCF are under operator manual control and other DCF are automated. 


(Sensing and Perception - SAP) 

The SAP subsystem is a video machine vision architecture Kiich provides the automation controller with th 
capability to track, acquire, and verify position/ orientation of aan-nade objects in real time. .» a pp r oaca is 
based an a set of computer vision algorithms and custom real-time video pro ces sor Much are described elsooen 
in these Proceedings by arian Wilcox. The c ur rent approach is limited to simple polyhedral objects* and use o: 
multi-camera monochranatic intensity imagery. Out-year development for the evolution domnstraticn of 199< 
targets a SAP subsystem with multi-sensor capability, soecif icaily. integrated use of intensity-raagi 
information. A ccmpieaentary goal is real-time capability for acquisition/verification of objects with ccmuec 
curved surfaces. 
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Smory and Conclusions 

NASA baa im ert a icen an aggressive progra m of telerobotica research and technology development. The b road 
goal of the overall NASA p r o g ra m is technology definition and proof -of -con cept in support of space servicing, 
assembly, and repair operations. The res ulting telerobotic capability baa potential to signiflcmtly impact 
space mission productivity, capability, and automation across a wide variety of scenarios. The NASA/ OAST 
Telerobotics P r o g r a m has the objective of providing a r es earch base in advanced automation technologies, and 
defining how these technologies can be synergistically integrated into a system architecture far evolutionary 
telerobotics. The pro gram is expected to contribute a variety of generic technology products in the areas of 
artificial intelligr ce. robotics, and hi— in factors and teleoperaticns. Examples of interest to both the space 
and terrestrial telerdbotics communities include: task planners and expe rt error diagnosticians; co n trol 
algorithms and real-time mechanization for multi-arm manipulation, as well as adaptive and redradant 
manipulation; cooperative articulation of robot mobility and manipulation; robust, full -DOF teleoperations 
controllers targetable to a wide variety of manipulators; effectors and accompanying modal displays; various 
rrm ind-display devices tailored to cognitive requirements of the huoan operator; and sensor-processor suites 
adapted to the particular domain requirements. 


The MASA/QAST p ro g r a m has the br o ader systems objective of defining a cohesive architecture for advanced 
supervisory autcmaticn of space servicing tasks. Sncrma is to be rtmmenfari through a series of ground-based 
breadboard concep t d— cnstrations tciich show an evolving capability far a ddres s ing increased task complexity, 
uncertainty, and autcmaticn. This work is hosted in the JPL telerobot testbed and draws upon the p r ogra m core 
research for its supporting subsystem technologies. Productive work in this are e is expected to have high payoff 
not only to space telerobotics, but to br oa de r NASA applications, and beyond. Fu nda m ent al Issues in supervised 
automation are being addressed: 1) Identification of realistic run-time allocation of sensing, control, and 
performance monitoring functions between a hunan ana machine for co^lex tasks; 2) definition of a resulting 
hierarchical shared control-and-ccmmunlcaticn architecture, particularly in a ddre ss ing management of u ncert a inty 
in sensor data and control strategy; and 3) the longer-term development and evaluation of supervised learning 
strategies for instantiation of desired task p r oc e dures and sequences. 
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1. Introduction 

The goal of building autonomous, intelligent systems is the major focus of both basic and 
applied research in artificial intelligence within NASA. This paper discusses the components 
that make up that goal and describes ongoing work at the NASA Ames Research Center to 
achieve the goal. 

NASA provides a unique environment for fostering the development of intelligent 
computational systems; the test domains, especially Space Station systems, represent both a well' 
defined need, and a marvelous series of increasingly more difficult problems for testing 
research ideas. One can see a clear progression of systems through research settings (both 
within and external to NASA) to space station testbeds to systems which actually fly on space 
station. 


2. The Long-Range Goal 

As a springboard for discussion, let us create a view of a "truly” autonomous space station 
intelligent system, responsible for a major portion of space station control (the exact system is 
unimportant). We will build a view of all of the functions this system should have, which of 
those functions we can achieve (nearly completely) today, which we can easily see happening in 
the next few years with some engineering-oriented applied research, and which will be doable 
only with substantial basic research (over at least the next five years). 

Our intelligent system will have full responsibility for a major functional component of 
space station; examples include power, communications, thermal management, and 
environmental control. It will be responsible for nominal control, acute and chronic failure 
discovery, diagnosis, and correction, and communications/cooperation with both interested 
humans (astronauts, ground controllers, scientists) and other intelligent computational systems. 
The following specific abilities are needed to be able to completely satisfy those 
responsibilities: 

• Scheduling of System Resources to Meet Utilization Requests — the ability to analyze 
tasks that involve the space station component being controlled in order to set up 
targets for resource utilization. 

• Real-Time Schedule Execution and Monitoring— the ability to translate task requests 
into executable system commands and to understand if the tasks have been 
adequately performed. 

• Dynamic Schedule Modification— the ability to change the resource utilization 
schedule to reflect both internal (poorly performed or understood tasks) and 
external constraints (new tasks added, conflicts with other space station systems, 
etc.). 

• Heuristic/Experiential Failure Detection and Diagnosis — the ability to utilize 
"shallow" knowledge based mainly upon prior (human or machine) experience with 
the space station system in order to notice and diagnosis problems with the system. 
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• Causal /Model-Based (Both Qualitative and Quantitative) Failure Diagnosis— the 
ability to utilize deeper, "first-principle'* knowledge to diagnose system problems. 

• Planning of Failure Corrections— the ability to determine a course of actions to 
repair a diagnosed failure. 

• Realtime Failure Correction and Monitoring— the ability to translate a correction 
plan into actions and understand if the problem has. indeed, been fixed. 

• Long-Term Trend Analysis— the ability to understand slow-to-develop trends either 
to prevent incipient failure or to adjust nominal control over a long period of time. 

• Explanation of Actions— the ability to explain, with clarity and brevity, all system 
actions, from resource utilization schedules to failure diagnosis to failure correction 
actions. Clarity and brevity imply different explanations for different classes of 
humans who wish to inteiact with the system. 

• Cooperation with Other Intelligent Systems— the ability to work in concert with 
other intelligent entities on space station, both human and computational. 

Four other necessary abilities cross many functional lines for our truly autonomous system. 
These are: 

• Reasoning Under Uncertainty— the ability to make sensible judgments and carry out 
reasonable actions when world knowledge is imprecise or incomplete, heuristics or 
models have built-in uncertainty, or actions have uncertain effects. 

• Learning— the ability to alter and improve all functionalities as conditions change 
and knowledge is added over time. Starting views of how systems operate in space 
may be wrong. Fault correction actions may alter system configurations. An 
autonomous system cannot remain static or performance will at best not improve or 
(more likely) at worst degrade to completely unacceptable. 

• Common Sense Reasoning— the ability to occasionally go beyond specific domain 
knowledge into broad areas of human experience. This includes the ability to 
bypass established reasoning mechanisms when unexpected events render them 
clearly useless. 

• Self-Understanding— the ability to understand: 

• When and how to utilize different functional abilities such as heuristic as 
opposed to model-based diagnostic reasoning. 

• When to act independently as opposed to asking for human or other 
computational assistance. 

• How to prioritize actions— when are things critical and when can they wait. 

. When a problem is beyond the system's range of understanding. 

• When new knowledge is worth saving either directly or as part of a new plan 
or piece of a model. 

Finally, to build such intelligent, autonomous systems, three pieces of artificial intelligence 
methodology become critical. These are: 

• Better Knowledge Acquisition— great improvement in our abilities to acquire, both 
initially and dynamically via sensor and other input, the heuristic and causal 
knowledge our system requires. In other words, tools and techniques for 
constructing the knowledge base. 
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• Managing of Large Knowledge Bases— how u> manage, optimize, check for internal 
consistency and completeness, etc, knowledge bases at least a few orders of 
magnitude larger than those used by current large systems. 

• Validation of Intelligent Systems— both practical and philosophical understanding of 
what it means to validate autonomous, computational systems. For space station 
systems, a mutual education and satisficing process with Space Station personnel and 
the aerospace industry will be part of the task. 


3. Where We Are 

The state of the art is marked by the three orthogonal areas of kinds of knowledge we can 
represent, tools for knowledge base construction and manipulation, and kinds of problem 
solving. Using one of the three representation paradigms of rules, frames, and logic, we have 
engineering solutions for storing a wide variety of beurisdc/procedural knowledge and 
factual/declarative knowledge. We have commercial tools (KEE, S.1, KnowledgeCraft and ART 
are the outstanding examples) which reduce the knowledge base construction. time by at least 
two orders of magnitude over raw LISP (or FORTRAN. PROLOG, etc. for that matter) and 
which provide a selection of inference methodologies. We can routinely carry out the entire 
knowledge-based system building task for structured selection problems using experiential 
knowledge, component configuration problems, fairly complex scheduling problems, simple 
planning problems, and 'intelligent front-ends” for abstruse modeling and database computer 
programs. 

In addition, we can represent and utilize a very limited amount of probabilistic or 
conditional knowledge about data and knowledge. Our systems can explain their chain of 
reasoning in simplistic textual and graphical forms. We can also construct quite sophisticated, 
personalized interfaces to knowledge-based systems. Finally, we can utilize precisely the same 
knowledge base for distinct purposes, including diagnosis, simulation, and training. 


4, What We Can Almost Do 

In several other areas, basic research is beginning to make the transition into engineering 
utilization. Specifically: 

• Causal models are beginning to be used on significant problems, augmenting and/or 
replacing experiential heuristics for diagnosis and fault correction. 

• Systems are moving from "leisurely” offline applications into realtime control 
systems (where realtime currently means on the order of seconds for evaluation and 
response time). 

• Systems that dynamically reconfigure plans to reflect changing conditions during 
plan execution are beginning to appear in real applications. 

• The blackboard framework for cooperative utilization of different sources of 
knowledge is becoming part of the inference "toolkit” This represents a "micro* 
view of distributed control among cooperating knowledge-based systems in most 
current applications. 

• Machine learning, still in very simple forms such as explanation based 
generalization, or learning by example, is starting to make an impact on fielded 
knowledge-based systems. 
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5. What We Need to Do 

Clearly there is an enormous amount of work to be done to take us from where we are (or 
soon will be) in the technology of building autonomous systems to a point where the long- 
range goal is satisfied. Just as dearly, ire at NASA Ames cannot do it alL However, the 
problem areas discussed in this document are relevant to most current work in artificial 
intelligence research, so we have a wide community of fellow researchers to draw upon. The 
following are the scientific and engineering research areas in which we at NASA Ames believe 
our involvement through internal work or support of external work makes sense over the next 
several years: 

• Reasoning Under Uncertainty— as discussed above, particularly focusing on 
integration into practical knowledge acquisition and repre s e ntation frameworks and 
demonstrations of utility. 

• Machine Learning— emphasizing automatic knowledge base expansion and correction 
as well as learning by discovery (carrying out sensor-based "experiments, etc.). 

• Causal Modeling and Simulation— particularly on methodologies for integrating 
these methods with both heuristic- based problem solving at one end and 
mathematical model based simulation at the other. 

• Next Generation Tools for Knowledge Acquisition. Representation, and 
Manipulation— developing better, faster, more versatile tools to "routinize" much of 
the knowledge engineering process. Of particular importance is getting past the 
knowledge acquisition bottleneck and removing a human knowledge engineer from 
the loop as much as possible. 

• Explanation and Interface Technology— making progress in communicating between 
knowledge-based systems and human users. We wish to add perspicuity and 
perspicacity of explanation, and make it much more easy to customize interfaces to 
individual preferences. 

• Constructing and Utilizing Large Knowledge Bases— fundamental experiments in 
treating very large collections of knowledge. 

• Acquisition of Design Knowledge and Data for Complex Systems— ensuring that the 
vast amount of information used and discovered during the design process not be 
lost for future knowledge-based systems that need to reason about the artifact that 
was constructed. 

• Advanced Methods for Plan Construction, Monitoring, and Modification — 
integration of current planning methods to ensure usability and flexibility within 
practical environments 

• Hierarchical Control of Multiple Knowledge- Based Systems— experiments in control 
of multiple systems by hierarchical levels of increasingly more general knowledge- 
based systems. 

• Distributed Cooperation among Multiple Knowledge-Based Systems— the alternate 
view to the previous topic coordinated control by cooperative information sharing 
through a common database or "blackboard." 

• Validation Methodologies for Knowledge-Based Systems— both low-level issues of 
software verification (eliminating redundancies, inconsistencies, etc.) and more 
fundamental issues of ensuring desired functionality. 

Note that certain topics relevant to the Long-Range Goal, in particular common sense 
reasoning and self-understanding, are not specifically covered in the above research topics. 
That is because, in the author’s view, meaningful work in those areas awaits fundamental 
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pr o gram in ocher anas like rwl modeling aad machine learning. There are armai very 
long-term remarch project s already und erway in the "mimingr raaearch areas (Lenafa CYC 
project at MCC is a good example) aad we will keep carefol track of thoae projects as part of 
oar overall un d erstandi ng of aat onomoe a s ystems. 


6. Getting It Done 

The previous section has established ambitious goals for oar research in intelligent systems. 
Over the next several years, we will be attempting to build a highly regarded basic research 
laboratory within the Artificial intelligence Research and Applications Branch at Ames As 
relevant personnel join the pr o gram , research projects in many of the above areas will 
commence. Indeed, as discerned below, several projects already exist or will begin in fiscal 
1987. How e ver, for the next year or so, oar greatest leverage will come from s ponsoring a 
modest number of extremely high quality external resear ch projects The external work, besides 
serving programmatic research needs will also, we believe, act to educate Ames personnel 
through collaborative discussions produce some short-term technology demonstrations and 
provide a stream of motivated students a significant number of whom we hope can be 
convinced to join NASA upon receipt of their advanced degrees In addition, we wilt provide 
attractive mechanisms for senior researchers to carry out portions of their NASA-sponsored 
work at the Ames Research Center through sabbaticals and summer visitations 

The final two sections of this document will briefly describe current work in intelligent 
systems in progress and sponsored by the Artificial Intelligence Research Branch at NASA 
Ames Research Center. 


6.1. Internal Research in Fiscal 1987 

Reasoning Under Uncertainty 

• Peter Cheeseman. John Stutz. Mary Duffy, et al. will continue their work in 
probabilistic reasoning with test applications in planning and learning The coming 
year will see a focus on NASA problem domains and comparisons with other 
schools of uncertain reasoning Long term goals revolve around developing 
engineering methodologies for routinely including uncertain reasoning in relevant 
applications across many different forms of problem solving 

Machine Learning 

• Michael Sims and Peter Friedtand will initiate a project on learning by discovery. 
The starting point will be Sims' work in mathematical discovery applied to the 
engineered systems such as those on Space Station. The eventual goal is to achieve 
self-improving knowledge- b as ed control and analysis systems. 

Causal Modeling and Simulation 

• William Erickson. Mary Schwartz, and Peter Friedtand will use the space station 
thermal system as a testbed for work in integrating causal modeling with 
experiential heuristics and mathematical models. A short term goal is to 
demonstrate practical accomplishments as part of the 1988 Systems Autonomy 
Demonstration Program. A somewhat longer-term goal is to formalize the 
techniques for determining when each of the different forms of reasoning will be 
most useful for difficult problems. 

Knowledge from Design Through Operations 

• Cecelia Sivard and Lilly Spirkovska will continue their work on capturing design 
knowledge on the solar photometer system. A short-term goal is a proof of utility 
of the methodology, by showing how knowledge acquired during the design process 
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can be effectively utilized during operations and maintenance of a copies device. 
A to m -term goal is bolding increasingly more sophisticated design tools that 
automatically co nst r u ct knowledg e bases useful during the entire design through 
operations pr occM , 

Advanced Planaiag Werk 

• Peter Cheeseman, ec al. will continue their work on dynamic scheduling and 
rescheduling. Peter Friedland will collaborate to see if skeletal planning 
methodologies can be integrated with the work. The eventual goal is to show 
realtime utility of the methodologies. 

Validation Methodologies 

• Peter Friedland and Carla Wong will begin a project in the practical aspects of 
knowle dge -b as ed system validation. This will include intense collaboration with 
Space Station and the space industry to prevent bottlenecks in the goal of a clear 
path to operational systems on space station. First results will be seen in the 1988 
Systems Autonomy demonstration. A longer-term goal will be NASA-wide accepted 
methodologies for knowl edge - bas ed system validation. 


6l2. External Research in Fiscal 1987 

Reasoning Under Uncertainty 

• Lotfi Zadeb. UC-Berkeley— Dr. Zadeh is the world leader in the branch of 
uncertain reasoning known as fuzzy logic During the coming year, we will begin 
the process of integrating his work with Ames-internal efforts. 

• Don Heckerman and Eric Horvitz. Knowledge Systems Laboratory. Stanford 
University— Heckerman and Horvitz are students who have become leaders in both 
theoretical and practical aspects of the uncertain reasoning work that originated in 
the MYCIN project. During the next year they will begin to determine the 
relevance of those methodologies to NASA domains. 

Machine Learning 

• Tom Mitchell and Jaime Carbone!!, Carnegie- Mel Ion University— Mitchell and 
Carbonell. are among the world’s leading authorities on research in machine 
learning. The work will emphasize explanation based generalization and learning by 
example and include both theoretical aspects and practical applications to NASA 
domains. In addition, both Mitchell and Carbonell are likely to spend significant 
amounts of time at the Ames Research Center helping build our internal strengths 
in machine learning. 

Advanced Planning Werk 

• David Atkinson. JPL— The Al research group at JPL will be conducting Ames- 
sponsored research in the integration of sensor-based planning, plan monitoring, 
and plan modification. This is the first effort in what we hope will be a long- 
term research collaboration with the JPL group. 

Hierachical Control of Multiple Knowledge- Based System 

• Ron Larsen. University of Maryland— We will continue to support Dr. Larsen's 
work on developing computational structures for hierarchical control of cooperating 
knowledge-based systems. His work is attempting to integrate traditional decision 
theory with heuristic and model-based control methods. 
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• Richard Volz. University of Michjpn Prof— or Voiz is carrying oat a variety 
projects in sensor-based learning, manipulation. and scene naderaanding related 
coo rdin a t ed control of robotic qitiunr. Tide research is being s po ns ored 
cooperation with the JPL-led Telerobotks Program. 

DbtrAoted Cbeperatisn aamag Multiple KawMgi-Ind Systems 

• Ed Feigenbatnn and Brace B u chana n. Knowledge Systems Laboratory. Stanford 
University — The Knowledge Systems Laboratory is conducting a significant new 
effort centering on methods for cooperative control among distributed expert 
systems. This work includes both lon g -term rrscsrch project s resetting in PhJX 
theses and short term development of MackboanMneed techniques for cooperative; 
distributed control. 

• Tom Sheridan. MIT— We have been supporting Professor Sheridan's work in various 
methodologies for distributed end cooperative control in robotic systems. This 
work will continue as a joint project with the JPL-led Telerobotics Program. 
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Space Telerobotic Systems: Applications and Concepts 

L. Jenkins 

NASA Johnson Space Center 
Houston, TX 77058 


1. Abstract 

The definition of a variety of assembly, servicing; and maintenance missions has 
led to the generation of a number of space tekrobot co nce p t*. The remote operation of 
a space telerobot is seen ss s means to increase astr onaut productivity. Dexterous 
manipulator arms are controlled from the Space Shuttle Orbiter cabin or a Space 
Station module. Concepts for the telerobotic work system have been developed by the 
Lyndon B. Johnson Space Center through contracts with the Grumman Aerospace 
Corporation and Martin Marietta Corporation. These studies defined a concept for a 
telerobot with extravehicular activity (EVA) astronaut equivalent capability that 
would be controlled from the Space Shuttle. An evolutionary development of the 
system is proposed as s means of incorporating technology advances. Early flight 
testing is seen as needed to address the uncertainties of robotic manipulation in space. 
Space robotics can be expected to spin off technology to terrestrial robots, particularly 
in hazardous and unstructured applications. 

2. Introduction 


Increased operations in space with the Space Station and the Strategic Defense Initiative define a need for remote 
operating systems to assist the space crews in accomplishing a variety of new functions. The role of the space crew is 
changing, with more missions recognizing the benefits of servicing and maintenance as a cost-effective mode of operating 
satellites. The size of the Space Station mandates its assembly in space. Other large space systems will require assembly on 
orbit. Recent Space Shuttle missions have demonstrated the effectiveness of the extravehicular activity (EVA) crew in 
many of the tasks needed for future space construction, assembly, and maintenance. As the magnitude of mission 
requirements grows, the productivity of the astronaut must be increased. The use of EVA is crew time intensive; it requires 
a buddy tystem as well as an observer in the cabin. Time spent preparing to leave the cabin, prebreathing oxygen, and 
maintaining equipment add to nonproductive time. Remote operating systems are a means of amplifying space crew output 
[1 ]. One concept for remote operations that has been defined in some detail is the telerobotic work system (TWS). 


The basic concept of the telerobotic work system consists of two dexterous manipulator arms controlled from a remote 
station (Figure 1). The direct control of the arms may be supplemented by interaction with a computer to perform certain 
tasks or portions of tasks. The tasks to be performed range from changing modules and components in the repair of 
satellites to the construction of large space systems like the Space Station. The operator is provided with sensory feedback 
of the environment and conditions at the work site. This approach is reflected in the term "telerobotics,'* which implies a 
combination of teleoperating and robotics. An objective of the system development approach is to increase the productivity 
of the operator through more robotic inodes havings higher degree of autonomy (2). 


A robot operating in the environment of space has analogies to a robot operating in hazardous or unstructured 
terrestrial situations. The development of a robot with the capability to operate in space can meet many of the requirements 
of terrestrial applications. Current NASA activities in telerobotics consist of studies and technology development at all 
centers. The Jet Propulsion Laboratory has a ground-based demonstrator, and Goddard Space Flight Center is the lead 
center for the flight telerobotic servicer for Space Station. 


3. System Requirements 

The functional requirements for the telerobot will be derived from the servicing of satellites; satellite repair, assembly, 
and construction; payload handling; and contingency repair of spacecraft. These functions may be further broken down into 
a variety of generic tasks. Examples of the tasks are removing and installing fasteners, connection of umbilicals and fluid 
lines, module replacement, and adjustment of thermal blankets. An operational consideration in the requirements is EVA 
equivalency. Space systems are being designed to interface with the proven capability of the astronaut in the 
extravehicular mobility unit. A telerobot with EVA equivalent capability can interface with the space system and also has 
an operational backup in the EVA astronaut. 



The performance of these tasks will be greatly affected by the environment The lack of gravity forces is the most 
significant effect on manipulative functions. Zero-g is beneficial in allowing large masses to be handled. Other sero-g 
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effects are less beneficial. Parts being handled are not positioned and oriented by gravity forces, and the free play in the 
joints of mechanisms becomes an uncertainty. The human factors that are imp a cted by the space-flight environment relate 
to the interaction with displays and controls. Posture is different in xero-g. Restraints will be needed for force-reflecting 
controllers, and visual perceptions may be distorted. 


4. System Architecture 


The major elements of the TWS concept are the telerobot, the control station, and the system processor. This configu- 


ration corresponds to the architecture for an automated system as defined by Holcomb and others [2] and shown in Figure 2. 
The robot interfaces with the remote site at which the mission functions or the state changes are to be accomplished. The 


control station is the operator's interface with the system through controls and displays. The system processor implements 
the operator's commands and directs feedback of the results. The potential of the architecture is illustrated in Figure 3 as 
an example of the relationships of functional components of the system. Those relationships may be defined as operator 
interface, task planning and reasoning, control execution, and sensing and perception. Effectively, there is an operator 
control loop, an executive control loop, and a local control loop at the remote site. These control loops provide feedback and 


interaction to enable accomplishing tasks in an effective and productive manner. 


5. Conceptual Designs of TWS 

The Lyndon B. Johnson Space Center has studied the TWS through contracts with Grumman Aerospace Corporation 
and Martin Marietta Corporation. Figure 1 illustrates the system arrangement and major components for the initial 
application on the Space Shuttle Orbiter. The system elements logically divide into the robot work station where the 
physical tasks are to be accomplished, the control station with the operator's displays and controls, and the system processor 
that provides the computer power and logic to make the system function. Mobility to reach the work site is achieved with 
the Shuttle remote manipulator system (SRMS). Stabilizer arms hold the TWS in position at the work site. Later 
applications of the TWS may achieve greater mobility by using a free-flying module similar to the manned maneuvering 
unit. 


The robot work station has manipulators and end effectors to perform physical tasks. Sensor suites monitor and 
measure conditions at the work site. Although work-site conditions in space are more structured than in many terrestrial 
situations, the configuration cannot be as well controlled as in most robotic uses in industry. The ability to determine the 
state of the task components is critical because of the inaccessibility in space; thus, the need for a preceptive and adaptive 
system. The concept of EVA equivalency is a strong driver in development of the configuration [3]. The capability of the 
EVA astronauts to perform dexterous tasks in the servicing and repair of satellites has been well demonstrated in recent 
Space Shuttle missions. Satellite designs are now being implemented in response to the demonstrated EVA capability. If 
the TWS can perform tasks equivalent to those of the suited astronaut, there will be satellites to work on. 


The EVA equivalency requirement has resulted in strongly anthropomorphic configurations in the contractor concepts 
[4 and 5]. Grumman has carried the human analogy one step further by using the acronym "SAM" for the Surrogate 
Astronaut Machine shown in Figure 4. The principal camera location responds to the operator's eye- to* hand relationship. 
Other cameras on the arms provide additional views of the task. Proximity, force feedback, and tactile sensing supplement 


the visual aids. A third arm functions to stabilize the TWS at the work site. A dexterous arm similar to the other arms is 
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proposed that would give some redundancy. The adequacy of a flexible arm to perform the stabilisation func tion may 
require flight testing. The most useful approach to end effectors for accomplishing tasks is the attachment of tools to the 
dexterous arum Tool stowage is behind "SAM’s" torso to reduce the volume of the manipulative system. 

Martin’s concept particularly differs from Gramman’s in the location of the tool stowage in the torso as shown in 
Figure 5. The d ex t er o us arms are seven-d egree of-freedom (7 DOF) electric drive manipulator arms. The stabiliser arm is 
proposed to be a stiller 5-DOF arm. The dexterous arms have a force-sensing wrist with an interchangeable tool device. 
This configuration allows use of special-purpose tools or a general-purpose gripper. Cameras and lights are mounted in a 
head with a 3- DOF neck. 

The operator interface at the control station is critical for effective interaction with the robot. Interior volume is at a 
premium in space, particularly on the Space Shuttle. For example, the design of the SRMS was driven to a resolved- rate 
control system bec aus e the swept volume to operate a replica master controller for such a long arm was difficult to 
accommodate in the Orbiter cabin. Six- DOF rate controllers are proposed by Grumman (Figure 6) Martin (Figure 7). 
Technique for controlling a 7-DOF arm is a technology development issue. Martin has suggested a hybrid control system 
that uses rate or pos i t ion depending on the task. Replica controllers to position the smaller dexterous arms are a potential 
tradeoff for TWS application. 



Figure 5. Martin work station concept 
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6. Program Development 

The development plan for the TWS is predicated on the need to increase the productivity of the crew; therefore, the 
plan is evolutionary in nature. In this logic, the TWS design must be capable of incorporating technology advances as they 
become available. This approach will depend on modular subsystems and precise definition of interfaces to enable the 
adoption of newer innovations. Another feature of the logic is the evolutionary route of teleoperation to telepresence to 
supervisory control to supervised adaptive robotics [2]. The implications of this approach are evident in the selection of 
feedback sensors that will be compatible with expert systems and artificial intelligence needed for adaptive robotics. The 
telerobot technology program of the NASA Office of Aeronautics and Space Technology is consistent with this development 
approach. In addition to the ground demonstration telerobot at the NASA Jet Propulsion Laboratory, a protoflight testbed 
has been proposed to support research and technology experiments, to validate ground simulations, and to demonstrate the 
utility of a dexterous manipulation capability for remote operations in space. The flight telerobotic servicer program for 
Space Station is currently being defined. The anticipated result is a telerobotic system that will have application on the 
Space Station, but will have been developed and demonstrated on the Space Shuttle. 




7. Summary 

The development of a telerobotic murk system or a similar concept represents a valuable resource for performing a 
variety of tasks in the unstructured and hazardous environment of space. Development and demonstration in flight test oa 
the Space Shuttle can lead to applications on the Space Station for the mobile remote manipulator system, the satellite 
servicer, and the orbital maneuvering vehicle. A system meeting these requirements can be of great use in developing the 
technology needed for many terrestrial applications of telerobots. Telerobots will find uses in personal service functions for 
disabled and aged people and in hazardous situations such as are found in construction and agriculture. 
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Abstract 


C ? 


Robots have potential for increasing the value of man’s presence in space. 
Some categories with potential benefit are: 1) performing extravehicular tasks 

like satellite and station servicing, 2) supporting the science mission of the 
station by manipulating experiment tasks, and 3) performing intravehicular 
activities which would be boring, tedious, exacting, or otherwise unpleasant for I 
astronauts. 

An important issue in space robotics is selection of an appropriate levey 
of autonomy. In broad terms we ... can^jd^fine three levels of autonomy/ 

1) teleoperated - an operator explicitly controls robot movement, 2) telerobotic 
- an operator controls the robot directly, but by high-level commands, without, 
for example, detailed control of trajectories, and 3) autonomous - an operator 
supplies a single high-level command, the robot does all necessary task 
sequencing and planning to satisfy the command. 

* 

»e chose three projects for our exploration of technology and implementa- 
tion issues in space robots, one each of the three application areas, each with 
a different level of autonomy. The projects were: 




satellite servicing - teleoperated 
, laboratory assistant - telerobotic 
y on-orbit inventory manager - autonomous 


r, 


CUe -uC ti t fj 


This p appr dgggrihPT projects ''and -gumma ri some results of 


testing 


1.0 Introduction / 

The Space station is intended primarily to be a facility for the advance- 
ment of science. It is not possible tp/ predict precisely the future course of 
scientific discovery. But, based on our historical experience, we can expect 
that when new tools, methods, and /the proper mind set are combined, then 
progress and discovery will follow. The Space Station provides such an 
opportunity. Scientists of the future will make these discoveries to the extent 
that they have time to immerse themselves in scientific problems. The key to 
making the Space Station valuable for science is ensuring that those scientists 
on orbit do not spend all their time in station keeping but on science. We view 
robotics as a major tool for accomplishing the science mission of the station by 
making crew time available for science and by automating part of the science 
task itself. 


This paper presents an overview of three robotics system test projects 
conducted in the Boeing Space Station Robotics Laboratory in Huntsville. 
Assembly and test were conducted with subcontractor support from Essex 
Corporation, Georgia Tech Research Institute, Transitions Research Corporation, 
and Westinghouse Electric Corporation. 


2.0 Component Technology and Integration Issues 

Our overriding theme in this work was integration of systems having 
embedded robots for applications useful in space. The idea of focusing on 
systems may seem paradoxical, but the problems of integrating the robotic system 
were more important to us than the component technologies. Of course, we were 
obliged to address the component technologies to make the systems work. One 
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interesting system-level descriptor of a robotic application is its level of 
autonomy. The level of autonomy of a robot can be thought of in two ways: one, 
the more autonomous a robot is, the more abstract are the instructions which 
operate it and, two, the longer it can act without human intervention. For our 
own convenience we defined three levels on the autonomy continuum: , 

a. Teleoperated - an operator directly controls each action of the robot. 
Typical control devices include joystick, teach pendant, and master-slave 
device. 


b. Telerobotic - an operator controls the robot by giving medium level 
commands like, "move to position A" or "load sample #2." 

c. Autonomous - the operator makes a single request for a logically 
complete service. The robot does whatever planning, obstacle avoidance, etc., 
which are necessary to complete the service. 

We wanted to explore these three levels of autonomy and transition paths 
for systems with low autonomy to higher autonomy. 

The component technologies and issues we wanted to address by incorporation 
into our systems were: teleoperation time delay, compliance, man-machine 
interface, sensors, robot control language, multi-arm control, video/lighting, 
vision, end effectors, and hand controllers. 

3.0 Selected Robot Projects 

Based on our systems interests and relevant component technologies, we 
selected three projects. Each project was chosen to be at a different level of 
autonomy. The component technologies were shared among the projects in such a 
way as to permit exploration of those technologies but without burdening any one 
project with too many integration problems. Figure 1 shows the allocation of 
component technologies and level of automation to our selected projects. 
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Figure 1 Allocation of Technology Issues to Projects 


The robotic systems were assembled in our Huntsville Space Station 
Laboratory. Figure 2 shows the laboratory floor plan. It consists of two major 
floor areas: a staging area for robot systems and an acoustically isolated 
control room. 
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Figure 2 Boexng-Huntsvillc Space Station Robotics Laboratory 


The staging area is equipped with a variety of special lighting types and 
black stage curtains. The workstation is located in the control room. A window 
between the two areas permits viewing during test setup but is equipped with a 
blind for visual isolation during testing. 

3.1 Teleoperator for On-Orbit Servicing 

This system provides one-g simulation of a teleoperated robot for 
characteristic on-orbit servicing tasks. Our objectives in this task were to 
drive out servicer and human operation design requirements. Figure 3 shows the 
general test configuration. The servicing tasks were represented on a 
half-scale satellite mock-up with simulated Orbital Replacement Units (ORU's): 
Flight Guidance Equipment, Rate Sensing Unit, Battery, and Hultimission Modular 
Spacecraft Module (MMS) . These replaceable units are attached to the mock-up by 
different space-type fasteners and in different orientations. Obstacles can be 
located to increase difficulty of changeout. The robot arm is a Unimation Puma 
560. A black and white CCD camera is attached to the robot wrist (see 
figure 4) . Two other color cameras with remotely operable pan-tilt-zoom-focus 
are placed for worksite viewing. Several end effectors were developed for this 
test: MMS tool, gripper, hexdriver, scissors, and insulation holder. The end 
effectors are electrically driven and mate to a common interface. For this test 
the workstation is configured with three video monitors, robot control screen, 
and keyboard voice-driven video controllers and robot controllers. 

Six Boeing employees were chosen as subjects in the human factors testing. 
They had no prior experience with manipulator controls. All had normal or 
corrected visual acuity as determined by a Class II Flight Visual Examination. 
Two Skylab astronauts also participated. 

The first testing stage was used to develop learning curve data and to 
compare user preference for teach pendant and joystick. Subjects completed 
approximately 400 runs with 80 hours of testing. Figure 5 gives an example of 
learning curve data. The subjects unanimously selected the joysticks over teach 
pendant as the controller they would choose to do a difficult task. The data 
shoved that subjects learned more and produced faster times with joysticks. 
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Figure 5 Example Individual Learning Curve 


The second testing stage was used to assess the impact of communication 
time delay on task performance. By the time we started this testing, the 
subjects each had a minimum of thirteen hours of manipulator control experience. 
Only the joysticks were used. Figure 6 lists the tasks and subtasks we used. 
As an example. Figure 7 shows the J-hook fastener used in one of the tasks. 
Figure 8 shows how task completion time and arm movement time depended upon time 
delay. 


TASKS 


Tuk 

Sub cask 

J-Hook 

o 

Looses Ch« hex bole 


0 

Unlatch the J-hook 


o 

Letch the J-hook 


o 

Tighten the hex bolt 


0 

let urn the m to the start position 

Erive Screw 

0 

Fleet of the tool on the drive screw 


o 

Engage the connector 


o 

Disengage the connector 


o 

ketum the are to the start position 

FGE OKU *e»oval 

0 

Grasp the EVA Handhold 


o 

lesrn the OVU 


o 

tecum the arm to the start position 


Figure 6 Tasks Used in Time-Delayed Teleoperation 
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Figure: 7 Example Task - J-Hoo« Fastener 



Figure 8 Effect of Time Delay on Task Time and Arm Movement Time 


TASK TIME 
ARM MOVE TIME 


After our teleoperation testing was complete, we vere given the opportunity 
to install the teleoperator on a Space Station Module Mock-up. In this case the 
task was to evaluate control of the teleoperator from an Elesent Control 
Workstation in a debris shield inspection Node. 


40 




Fxcime 9 Teleromtxc Lmoratocv Assistant 


3.2 Telerobotie laboratory Assistant 

Our teleoperation work in the previous section showed that many manipula- 
tive tasks can be performed under direct control even with a significant time 
delay. Direct teleoperation should not, however, be considered an acceptable 
long-term solution. Two important factors score against it: The cumulative 
risk of damage is high and a highly skilled human operator is required. 

This application addressed the use of a robot in telescience. Astronauts 
will most usually be skilled generalists. We cannot, however, expect then to be 
experts in all aspects of Space station sciences. One way to get best use out 
of the Space Station is to permit a ground-based scientist to conduct his 
experiment on orbit with the feeling of "being there." This project integrates 
a telescience workstation with process control of a chemical vapor transport 
(CTV) furnace, mechanical control of embedded experiment automation, and a 
laboratory assistant robot 

Figure 9 shows the general arrangement for this project. The robot control 
is at a higher evolutionary level than in our direct teleoperator work. A 
hierarchical control structure has been built up from very low-level cossands. 
Error conditions are sensed and handled at the lowest practical level. Condi- 
tions which cannot be handled by the system are presented to the operator for 
resolution. 

A configuration schematic is shown in Figure 10. The Unimate standard 
programming language VAL II was used. However, some special-purpose routines 
were coded in assembly language to facilitate sensor and command port communica- 
tions. Ultimately, movement of an object was achieved with a VAL MOVE command. 
He implemented several ways to move an object: normal, shielded, constrained, 
or compliant. 

Shielded motion helps to prevent damage to a robot and its payload. This 
"force shield" stops motion if forces and torques measured by the wrist sensor 
exceed expected threshold levels. Threshold levels might be exceeded if, for 
example, the tool or payload comes in contact with an unexpected surface. 

Constrained motion involves shielded motion to the vicinity of a target 
point and additional movements until expected force and torque values are 
reached . 

Compliant motion uses force/torque feedback to complete an action such as 
insertion of a tool into a hole. If a threshold is exceeded using complaint 
motion, but it is not the success constraint value, then attempts are made to 
comply by altering the robot path based on direction of the exoerienced forces. 
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Figure 10 Configuration Description for Laboratory Assistant 


Additional control primitives allow for object alignment and slip detection 
using a tactile sensor pad in the gripper. These control primitives were 
combined in a hierarchical fashion to permit commands like: "grip selected 
ampoule,” "insert ampoule,” etc. 

Remote operation of a CVT crystal growth experiment was simulated by 
placing the CVT equipment rack within reach of the robot and by placing control 
computers and monitors in the adjacent control room. The telescience work- 
station contained three video monitors: two were normally timed to view the work 
area and one to monitor crystal growth. 

The scenario began by the robot attempting to find a reference location on 
the rack. This was a small post on the rack front surface. The robot grasped 
the post and used the force/torque sensor to fine-tune the location of the post 
relative to the robot. The CVT system was initialized by moving the ampoule 
positioning mechanism to a home location so that an ampoule could be loaded 
without coming into contact with the furnace. The robot was then commanded to 
grip a selected ampoule, insert into the positioner, and then to release the 
ampoule. The positioner was then commanded to insert the ampoule into the 
furnace. The predefined furnace temperature and ampoule positions were then 
executed. The telescientist viewed crystal growth with the microscope-video 
system, changing temperature and position parameters as necessary. When the 
crystal growth was complete, a sequence of commands was made to remove the 
ampoule, place in a cooling rack, and stow for shipping. 

3.3 Autonomous Logistic Robot 

The two projects described thus far each required more or less continuous 
human supervision. Certainly supervision in the telescience case was at a 
higher level, but there was no intent to ignore the system until the job 
completion. In this project our goal was to integrate a system which would 
complete an entire task without human guidance or intervention. In order to 
make this possible, we chose a rather structured environment: a mock-up of a 
Space Station logistic module. We did not fully structure the problem space in 
that we did not require objects to be precisely located. Figure 11 shows the 
general system arrangement. Figure 12 shows the equipment configuration. 
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Figure 11 Two-Armed Logistic Robot 




Figure 12 Logistic Robot Equipment Configuration 
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The. logistic module segment mock-up of a radial configuration was approxi- 
mately half-scale. The drawers and racks were mounted so that articles would 
fall if not constrained. For test purposes, a set of simple tools was used to 
represent typical on-orbit inventory items. The purpose of the rail-based 
transporter was to grossly position the robot in the area of its tazget and to 
allow the robots by means of force and vision to accomplish its task. 

The robot consisted of two Puma 560 arms mounted on a transporter carriage. 
The arms shared a common work space. Arm-to-arm messages were used to 
synchronize arm movements. 

Each arm had a different gripper. One had an integral barcode reader. It 
was primarily for identifying drawers and racks and for grasping handles. The 
other hand had a servo gripper which was equipped with automatically inter- 
changeable fingers: one set for gripping handles, the other for picking up 
small objects. The servo feature allowed measurement of position velocity and 
force parameters. 

The Automatix AV-5 vision system was used to identify items in a drawer and 
to locate open areas where they could be placed. The AV-5 also served as system 
coordinator. 

Test scenarios have allowed us to automatically identify, store, and 
retrieve inventory items, racks, and drawers. Two-arm activities are 
coordinated at least to a collision prevention level. Removal of a full rack 
was a two-armed activity and required fully cooperative two-armed motion. 

4 . 0 General Observations 

In order to build systems with embedded robots, we found it necessary to 
use several different computers, operating systems, and programming languages - 
not that it was a surprise, but it was a nuisance. If the level to which 
standards are used in a discipline is one measure of its maturity, then robotics 
is immature. We need to apply some standards which will permit interconnection 
and cooperative use of diverse equipment (e.g., vision, arms, grippers, etc.). 
A layering standard along the lines of the IS0-0SI definitions is essential for 
design of reusable components and evolving robot autonomy. We don't want to 
stifle research, but those who must build real systems for space will be greatly 
aided by standards. 

Our teleoperator servicing tests were based on existing satellite design. 
It didn't take us long to decide that existing satellites would be extremely 
difficult to service by robot and that two arms would be required. A review of 
the manipulative difficulties also showed that those tasks could have been 
designed for relatively simple one-arm operation. The systems we expect to 
manipulate in space should be designed for robotic attention. Our industrial 
experience tells us that a task designed for robots can be performed better by 
humans . 

Our time-delay work showed that typical tasks can be done by teleoperation. 
They just take longer with time delay. The situation is really a bit more 
complicated. Our test subjects were not repairing a multimillion-dollar 
satellite. There was a test observer with a panic button carefully watching for 
the robot to get into trouble. In fact, a significant number of runs were 
aborted to protect equipment or to release a jammed tool. 
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Telerobodc Assembly of Space Station Truss Structure 

G. Fischer 

Grumman Space Systems Division 
Bethpage, NY 11714 


1. Abstract 

This paper discusses methods of assembling the Space Station’s structure which only utilize telerobodc devices 

• an approximately anthropomorphic telerobot with two dextrous arms 

• the Shuttle Remote Manipulator System (SRMS) 

• various material handling machines. 

Timelines and task recommendations for autonomous operations are also included. The paper also describes some 
experimental results comparing two manipulator control devices. 



2. Introduction 

Recent studies at Grumman have shown the feasibility of assembling the Space Station truss structure in space 
using only telerobodc systems. The studies investigated the use of a pair of cooperating dextrous manipulators in at 
anthropomorphic telerobodc system. Two types of investigations were conducted: 

• Computer Aided Design (CAD) studies of the assembly of the Space Station truss structure 

• Experimental tests of several subjects and control devices performing some truss assembly tasks using a 
pair of dextrous manipulators. 

These studies were based upon the capabilities of current state -of-the art electro-mechanical devices. Although 
human directed telepresence control was the baseline of these investigations, the trass assembly activities described 
here lend themselves very nicely to autonomous (or supervised) robotic operations. This paper will present some of 
the results from both the CAD studies and the experimental investigations. 

The CAD studies addressed the problem of assembling the entire Space Station truss structure from the Cargo 
Bay of the STS Orbiter. without using Extra \fehicular Activity (EVA). The studies only addressed structural 
connections and did not consider the installation of utility lines (fluid and electrical). Two different methods of 
assembly were explored. Both utilized an anthropomorphic machine known as SAM (Surrogate Astronaut Machine) 
to perform all the dextrous manipulation tasks needed in vacuum to assemble the structure. Figure 1 displays some 
of SAM’s characteristics. The “third arm’’, which normally functions as a means of attaching SAM to a worksite 
to allow the SAM mobility aid (SRMS or MSC) to depart and perform other functions, was not utilized in this 
study. This paper reports the features and significant differences (including task timelines) of the two assembly 
methods. 

A number of experimental results were obtained from a series of test subjects operating a pair of six 
degree -of-freedom (6DOF) manipulator arms in a telepresence mode. Through the use of voice controlled cameras, 
the operators relied on video and force feedback to retrieve a “strut” and connect it to a truss node (see Figure 2). 
Two different control devices were used by each operator 
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Figure 1. Astronaut Capabilities With SAM 



Figure 2. SS Structure Assembly Tests 


• A pair of 6 DOF bail type hand controllers utilized a resolved rate control law (see Figure 3). 

Some force feedback was provided by auditory signals. No force feedback was provided through 
the ball controller. 

• A pair of bilateral force reflecting tBFR) replica master controllers utilized position-position error 
signals tc produce forces in both master and slave arms (see Figure 3). 

This paper reports the significant differences found in operating with both of these control systems. 

3. Structure Assembly Tests 

Three struts were assembled into a node which contained strut termination fittings (see Figure 2). The strut 
connections, known as the ‘ * Wendel- Wendel ’ ’ joints, require that one manipulator hand hold a strut in position 
while the other manipulator hand translates a collar over the joint and then locks the joint by rotating the collar 
about half a revolution. The struts were positioned in the nodes in three orientations: vertical, diagonal and 
horizontal. In a gravity field, task difficulty was strongly influenced by stmt orientation. The vertical strut 
installation was very easy to do. The horizontal strut installation was Quite difficult. Task times were recorded from 
the start of stmt removal from the vertical storage (the left zone of Figure 2) until the visual indicator on the stmt 
locking collar indicated a locked condition. 



BFR MASTER 
CONTROLLER 



Figure 3. Control Station 


Figure 3 shows the two types of control devices which were utilized for these tests. The Master Controllers are 6 
DOF BFR arms which have an additional DOF squeeze grip for operating parallel jaw motion end effectors on the 
slave arm. The master and slave arms have identical structures and kinematics (i.e.. a geometry ratio of 1:1). The 
control laws used by this BFR replica system develop torques, at both the master and slave joints, which arc 
proportional to the position error signal between the corresponding master and slave joints. That is, when the 
master elbow is displaced 30° with respect to the slave elbow, the operator feels a force at the control handle 
(which was generated at the master elbow) which tends to drive the master arm to the same position as the slave 
arm. Simultaneously, the slave arm experiences a torque at its elbow which tends to drive the slave arm to the same 
position as the master. Thus, high forces at the slave arm are experienced by the operator as high forces on the 
master arm. This type of control system is known as bilateral force reflection (BFR). 

The second type of manipulator controller shown in Figure 3 is a 6 DOF ball gripper type which is under 
development by CAE Electronics. Ltd. This compact device, and its supporting electronics, translate operator^ 
displacement commands ( + - x. y. z. 6 , <t>* ^) at the ball grip into slave end effector rate commands (+ - x, y. z, 
0, 0) in one of several (selectable) coordinate axis systems. For these tests, the selected system fixed the slave 

hand 0, <t> a vb $ axes to the slave hand. and. when the hand grabbled a strut or collar, to the work object. Thus, 
ball grip angular input commands were in a work object coordinate system. Translation commands (x. y, z) were 
always in an inertially fixed reference frame. Thus, plus z was always straight up. Operator forces at the ball grip 
were very light (with no felt feedback from the slave arm). An auditory system supplied some indication of high 
forces on the slave arm. This auditory system was not very helpful for manipulator operators of this test series. 

Figure 4 shows the laboratory in which these tests were conducted. The worksite region is on the right side. The 
operator region is on the left side. An opaque curtain was placed between these regions tor these tests. The 
operator received all visual information from three TV monitors (see Figure 3). These monitors received images 
from three fixed location cameras in the worksite region. The cameras had 3 DOF (scan. tilt, and zoom) and were 
controlled by manipulator operator voice commands. 

One of the major objectives of this test program was to evaluate the effects of the two manipulator control 
systems (master vs ball controller) on task timelines and to identify benefits and problems associated with them. 

The results we found were: 

• 230% faster strut installation with the master controllers 

-least time difference for the vertical strut, w hich required the lowest cognitive workload of the ball controller 
tasks. 
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Figure 4. Test Lab Overview 
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These, and other, experimental results are reported in greater detail in Reference # 1 . 

4. Space Station Assembly Study 

Two different methods of using telerobotic devices (SAM) for assembling the Fall 1986 Langley Task Force 
Space Station Design were explored. Both methods used SAM in an operator controlled telepresence mode. The 
major differences between the two methods were in how SAM moved around the worksite and the amount of 
automation used to enable SAM to obtain supplies for the construction activity. 

• Method 1 • Method 2 

-2 Telepresence controlled SAMs -1 telepresence controlled SAM on SRMS 

-Rotating/Translating Fixture Tool for SAM -Rotating SS assy fixture 

-Automatic strut & node dispensers 
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Figure 5 depicts a partially completed Space Station truss structure emerging from an assembly fixture at die 
back of the Orbitcr’s Cargo Bay. The truss is formed of cubical “bays” which are 5 meters long. Each face of a 
cube has a diagonal strut in addition to “horizontal” and “vertical” struts. Figure 6 shows more details of the 
support of the Truss Assembly Fixture within the Orbiter’s Cargo Bay. The truss is assembled in the lower bay 
region by a SAM attached to a “horizontal” beam. The beam is attached to a turntable which provides 2 DOF: 
translation in the “vertical” (Orbiter z axis) direction and 360° rotation about this “vertical” axis. These 2 DOF. 
and the X) DOF within SAM. allow SAM to reach all comers of the lower bay and the storage regions for struts 
and nodes (which are located outside the truss at the midpoints of the lower horizontal face of the cube). When the 
truss dement feed system canisters become empty, they are replaced with foil canisters by a second SAM which is 
mounted on a Shuttle Remote Manipulator System (SRMS). The canisters contain mechanisms which deliver all 
truss elements (i.e., struts and nodes) to the same location within a canister to expedite assembly operations. 







Figure 5. Truss Assembly Fixture, 
Method 1 





NODE FEED SYSTEM CAMSTER 
• RESUPPUED BY 2ND SAM ON RIBS 


[ /turmtabif DIAGONAL TRUSS , 

\s TURNTABLE PEED SYSTBI CAMSTER 

/ • RESUPPUEDBY 


2ND SAM OM RMS 


LONGERON TRUSS 
FEED SYSTEM CANISTER 
• RESUPPUEDBY 
2ND SAM ON RMS 


MPESS 


Figure 6. Truss Assembly Fixture Details 


All Space Station structural assembly sub-tasks have been considered for telerobotic assembly. Reference 92 
contains a listing of these sub-tasks and our estimates for the time required for their completion. These sub-task 
times were grouped into major task activities and summed for the entire operation of assembling the structure of the 
Space Station’s Transverse Boom, which required 70 hours of on-orbit time. These data are displayed on Figure 7. 
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Figure 7. Combined SS Assembly Operations Timeline, Method 1 
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Wfe examined a second method of assembling the Space Station’s Transverse Boom structure. This method only 
used a single SAM which remained attached to the SRMS. The partially constructed SS structure was rotated about 
the centerline of the assembly fixture (Orbiter z axis) to locale all tefeoperations in die same region of the Orbiter. 
This region allowed easy access to strut and node storage areas within the Cargo Bay and allowed SRMS motion 
from top to bottom of the assembly fixture. Figure 8 sbdfts part of the fixture assembly operation: the upper 
portion of one of the four truss supports is about to be inserted by SAM into the previously installed lower two 
pieces. Figure 9 depicts a node receptacle (which contains 6 nodes) installation into the assembly fixture. Figure K) 
shows an assembled truss bay above four previously assembled “horizontal” truss faces within the assembly 
fixture. SAM is positioned to raise the completed bay to the top of the fixture where it will be held by the fixture. 
The second bay is completed by SAM attaching vertical and diagonal struts between the completed bay and the next 
“horizontal” truss face, which has to be raised up to the bottom position. During vertical face assembly, the 
partially completed structure is rotated 90° to present SAM with a new comer of the truss. During this 90° 
rotation, angular accelerations have been limited to 1 “g”. This limits bending moments on die cantilevered 
diagonal struts (since they are attached at only one end during rotation) to values winch are substantially below their 
strength. The six timelines associated with assembly Method 2 are contained in Reference #2. The last of these is 
reproduced here as Figure U. Method 2 required 72 hours of on-orbit time to assemble the Space Station’s 
Transverse Boom structure (work began after 8 hours on orbit). 


90* INCREMENTAL ROTATIONS 



Figure 8. Installation of Assembly Fixture. Method 2 



Figure 9. Details of Node Receptacle Installation 
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90° INCREMENTAL ROTATIONS 



Figure K). Installation of Vertical Face Struts 



| T«mi»AvscoMnmp }| ~ jgj [»| 


•• tf( OCTUUOTW( LNCOftASK 


Figure 11. SS Truss Assembly (Day 6) Timeline, Method 2 


5. Tasks for Autonomous Telerobotics 

Both of the assembly methods which are discussed in this paper have assumed that all motions of the telerobot 
(SAM) and the SRMS have been commanded by astronauts at control stations within the Orbiter. Since telerobot 
operations are planned for 16 hours per day, astronauts controlling these devices may experience fatigue, even with 
frequent shift changes. Consequently, the reduction of astronaut workload is desirable to reduce fatigue and the 
concomitant probability of errors. 

Selective autonomous telerobot actions can reduce astronaut fatigue by occasionally eliminating the need for an 
astronaut's physical effort and mental attention to details. At the completion of an autonomous robotic task, the 
astronaut acts like a supervisor and verifies that the autonomous task was performed properly. 

Selective autonomy is stressed because of the prohibitive costs of providing a telerobot which is capable of 
performing all Space Station assembly tasks in an autonomous mode. Also, to perform autonomous tasks in a 
manner which does not impose a significant weight burden on the Space Station, a high level of machine 
intelligence is required. At this time, one can not predict that this machine intelligence technology will be available 
in time to perform the complete assembly of the Space Station. 
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Our work has persuaded us that the Space Station structure can be reliably assembled by teleroboOc systems. We 
believe 

• that the majority of tasks should be under astronaut control using telepresence technology 

• -bat selective tasks should be performed autonomously by the tekrobot 

• that the primary telepresence control device should be a bilateral force reflecting replica master 

• that the initial Space Station structure can be assembled during 1 STS mission without EVA. 

The work described in this paper was performed under contract to the National Aeronautics and Space 
Administration. Johnson Space Center. Testing was conducted at Grumman's Manipulator Development Laboratory. 

I would like to thank Mr. Erik Erikscn and Dr. John O'Hara for their contributions to our test programs, and. 

Mr. Dan Guinan and Mr. Kirk Sneddon for their Space Station assembly studies. 
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1. Abstract 




Semi-autonomous inspection and servicing of Space Station’s major thermal, electrical, 
mechanical subsystems is a critical need for the safe and reliable operation of the Station. A 
conceptual design is presented of a self-intelligent, small and highly mobile robotic platform. 
Equipped with suitable inspection sensors (cameras, ammonia detectors, etc.), this system’s pri- 
mary mission is to perform routine, autonomous inspection of the Station’s primary subsystems. 
Typical tasks include detection of leaks from thermal fluid or refueling lines, as well as detection 
of micro-meteroid damage to the primary structure. 

Equipped with stereo cameras and a dexterous manipulator, simple teleoperator repairs and 
small ORU changcout can also be accomplished. More difficult robotic repairs would be left to 
the larger, more sophisticated Mobile Remote Manipulator System (MRMS). An ancillary func- 
tion is to ferry crew members and equipment around the station. 

''Primary design objectives were to provide a flexible, but uncomplicated robotic platform. 
One which caused minimal impact to the design of the Station’s primary structure but could 
accept more advanced telerobotic technology as it evolves. 



Figure /. Manned Mobile Sen. .r 
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2. Introduction 

The Space Station will undoubtedly be the largest and one of the most complex “spacecraft* * 
ever launched by man. Bounded by a structural surface of more than 2 acres, the Station will 
contain many miles of electrical power, thermal fluid and data communication utility lines as well 
as house dozens of primary and secondary subsystems and components. Many of these elements 
have design lives of 20 years or more and must function reliably during this period in the hazard- 
ous environment of space. In recognition of the need to enhance the operational efficiency and 
reliability of Space Station, a congressionally appointed Advisory Committee 11], recommended 
that the initial Space Station should utilize a high degree of automation and robotics (A & R) 
technology. Among many of the Committee’s recommendations was a suggested NASA A & R 
demonstration to construct “a mobile *go-fer* robot to assist in crew tasks’’ [1]. The concept to 
be discussed in this investigation addresses this important recommendation. 

Considerable work has been performed, for example see [2 to 4], in identifying teleoperator/ 
robotic concepts and technology to assist in the on-orbit servicing and repair of spacecraft. It is 
clear that automated robotic work systems can considerably enhance the productivity of the flight 
crew. This is true, provided that the servicing tasks are well-defined, and secondly, that the 
required servicing mechanisms and the equipment to be serviced have been “scarred” to accom- 
modate such automation. Furthermore, hazardous tasks, such as a propellant refueling opera- 
tion. would obviously be more safely performed from a remote site. 

Man’s permanent presence onboard Space Station offers new and greater opportunities to 
repair and service in-orbit spacecraft. Robotic retrieval of satellites via free-flying robots (or 
robotic Orbiting Maneuverable Vehicles (OMV’S)) assisted by a teleoperated RMS are logical 
applications of A & R technology. However, considerable advancements in automation technol- 
ogy are still required, ranging from control architecture, task planning and artificial intelligence 
(AI) to robotic manipulator design and external sensor development [1]. Examples of how future 
space flight telerobots would differ from those in industry can be found in [5]. 


3. Space Station Inspection and Servicing 

Apart from servicing orbiting payloads and spacecraft, the complexity, size and longevity of 
the Station warrants extensive application of automation to perform the necessary “housekeep- 
ing” and maintenance functionr.-Representatrve examples of the station’s subsystems which wifi" 
require periodic inspection and/or servicing are illustrated in Figure 2. Fault detection and isola- 
tion will be needed for the electrical power cables, communication and data lines, and those used 
tor the thermal environmental control system. Detection of hazardous leaks from propulsion fuel 
lines or from the thermal fluid bus carrying anhydrous ammonia will also be a concern. 

Micro-meteroid damage to the primary or secondary structure, solar panels, radiators, etc. 
must also be checked. Environmental damage such as that due to long term exposure to atomic 
oxygen or UV radiation to structural materials may occur as well. Contamination of optical 
surfaces, mirrors, and array panel surfaces can be expected. The diagnostic/maintenance list is 
extensive. 

Reference [6] addresses many of those inspection and servicing needs of the Space Station 
from the standpoint of A & R. In this study, candidate A & R functions were identified, ranked 
and costed. Weighted assessments were made in terms of safety, productivity, IOC cost, risk, 
spinoff likelihood, reliability/maintainability and commonality. Table 1 is an example of one of 
the value ranking tables in [6], showing the priority of the first 29 of the 58 A & R candidate 
functions evaluated. It becomes apparent, upon reviewing this list that many of the inspection 
related tasks are not only important to perform but, moreover, have some of the most favorable 
cost-to-benefit ratios. For example, see utility run, truss/structure and thermal control system 
inspection items. 

Another important conclusion from the study performed in [6], is that a substantial savings 
in crew time could be realized by automating the inspection process. According to Figure 3, 
automating inspection activities represent a savings of 90 r « of the crew’s time relative to 10 r « for 
those due to repair. This finding is based on the realization that inspection related activities are 
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both more frequent and time consuming. This is not to infer that the inspection process is neces- 
sarily more important than repair. A repair to a critical system, while not necessarily time con- 
suming, could be critical for safe operation. 
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Figure 3. Inspection Versus Figure 4. Inspection Travel Distances 

Repair Robotic Tasks 

Some appreciation of the magnitude of the inspection process can be gained from Figure 4. 
Considerable crew effort is involved in just examining the 3.4 miles of tubular struts which com- 
prise the Station. Add to this the miles of electrical, thermal and data lines which could develop 
problems during the Station’s 20, 30 or 30 year life. Fortunately, most of the required inspection 
activities can be performed without the need for extensive crew EVA time by utilizing a special 
purpose robotic mobile platform in conjunction with internal system sensors. The Global Opera- 
tional Flight Iaspectlon/Repair System or “GOFIRS” can perform (alone or in conjunction with 
a robotic MRMS) many of the inspection/servicing tasks identified in Reference [6] as listed in 
Table 1 (see asterisk items). 

4. The “GOFIRS” Concept 


In establishing the conceptual design of a robotic platform to meet the Station’s needs, 
certain ground rules had to be established. These appear in Table 2. 

Table 2. Major Conceptual Ground Rules 


TASKS 


- AUTONOMOUS INSPECTION (“SCOUT* FOR MRMS/TRANSPORT) 

- ASSIST CREW EVA (TOOL ORU RETRIEVAL) 

- TELEOPERATED SERVICE & REPAB (SMALL ORU MAINTENANCE) 

- CREW TRANSPORT (WHEN REQUBED) 

- ACCOMODATE FTS (F COST EFFECTIVE) 


• PERFORMANCE 

- ACCESS STATION PRMARY SUBSYSTEMS 

- REQUIRE NO MAJOR MODIFICATIONS TO STATION STRUCTURE 

- ON-BOARD INTELLIGENCE FOR INSITU DIAGNOSTICS OF SENSOR DATA 

- MANIPULATOR WITH ROBOTIC SENSOR (STEREO VISION FOR TELEOPERATOR SERVICING) 

- REACH ANY POINT ON STATION WITHIN 5 MBMJTES (RATE-100 FT/MIN) 

- PROVIDE MINIMAL VIBRATION DISTURBANCES TO STATION 

- SATISFY ALL OTHER STATON OPERATIONAL. REL1ABUTY. & SAFETY REQUIREMENTS 


An important ground rule is that the GOFIRS is not intended to replace the much larger, 
more sophisticated M RMS/Transporter but to augment its capabilities by detecting possible mis- 
sion threatening defects or faults. The ability of MRMS to perform routine inspection of the 
myriad of subsystems on board Station is limited by its large size (spanning more than a 5 meter 
bay) and its relatively slow speed (less than 2 feet/minute). A small, highly maneuverable plat- 
form capable of accessing tight interior spots could be designed to be simple enough, hence 
affordable, so that several GOFIRS could be on continuous patrol. 
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Although its primary mission is one of inspection, crew EVA assistance and transport could 
also be provided. Small scale teleoperated servicing and repair could also be accomplished. This 
could be particularly valuable if the MRMS was tied up completing an activity on one end of the 
Station when some system needed immediate servicing cn the other end. 

Another ground rule is that the GOFIRS should make maximum use of current robotic 
technology (sensors, computer architecture, manipulators, etc.) and yet have sufficient growth 
capability to accept more advanced A & R technology as it comes on line. An example of this is 
the Flight Telerobotic Servicer (FTS) concept to be developed as a “robotic front end” to the 
MRMS and OMV. 

In terms of performance, the GOFIRS should be able to reach all or most of the important 
subsystems. It would be highly desirable to provide this mobility without disturbing the basic 
design of the primary structure by using miles of additional track or special cabling. GOFIRS 
should be equipped with sufficient onboard sensors and intelligence to perform routine analyses 
of inspection data and report anomalies and their location back to the command module. For 
example, the location of a leak in the thermal fluid lines would be identified. An onboard micro- 
processor would make a determination of the extent of this leak. Based on preprogrammed lim- 
its, immediate crew attention could be requested, or the anomaly could be simply “logged” for 
the next schedule maintenance activity. 

Clearly, deciding a course of action based on real time sensory inputs would embrace the new 
and growing technology of Artificial Intelligence or AI. If corrective action is needed, a GOFIRS 
* equipped with the appropriate teleoperator/telepresence sensors, cameras and manipulator, 
could make the repair under the control of a human operator. If the repair could not be made 
remotely from the command module, then the GOFIRS could transport a crew member to the site 
to make the repair. In a more futuristic version of this scenario, the repair could be made autono- 
mously by GOFIRS under the automatic control of an “expert system”. 

Due to its mobility, GOFIRS would offer a secondary benefit of being able to ferry the crew 
and needed equipment, tools or ORU’s around the Station for EVA. A minimum nominal travel 
rate of 100 feet/minute (1.1 miles/hr.) would enable any point on Station to be reached within S 
minutes. Of course the mass and acceleration rates must be sufficiently small as not to induce 
significant vibrational disturbances into the Station. 


Table 3. The “GOFIRS” Concept 
GLOBAL OPERATIONAL FLIGHT INSPECTION REPAIR SYSTEM 

• SELF-INTELLIGENT, SMALL. HIGHLY MOBILE INSPECTION & ROBOTIC REPAIR PLATFORM 

- WHEEL (OR MAST) DRIVEN. TABLE-TOP SIZED VEHICLE WHICH CAN ACCESS MAJOR 
AMD MINOR SPACE STATION SUBSYSTEMS. 

- EQUIPPED WITH ON-BOARD SENSORS & TELEMETRY TO AUTONOMOUSLY PERFORM 
ROUTINE INSPECTION. 

- EQUIPPED WITH CAMERAS A MULTI-DEGREE OF FREEDOM ROBOTIC MAMPULATOR TO 
PERFORM EITHER TELEOPERATOR OR TELEROBOTIC SERVICING OR UTUZE FTS AND ORU'S 

- PROGRAMMABLE MICROPROCESSOR TO PERFORM ASSIGNED TASK. TRACK CURRENT 
LOCATION AND PERFORM ROUTIfC ANALYS5 OF INSPECTION SENSOR OATA. 

• MULTIPLE “GOFIRS" CAN CONTINUOUSLY PATROL DESIGNATED SEGMENTS OF SPACE STATION. 

• CAN ASSIST MSC/TRANSPORTER AND ASTRO NOUTS IN EVA ACTIVITIES. 

• CAN FERRY CREW FROM POINT TO POINT 

• “LOW LEVEL" GOFIRS CAN BE DEVELOPED IN THE TIME FOR FLIGHT EXPERIMENT DEMONSTRATION. 


Features of the GOFIRS concept which meet the above ground rules are summarized in 
Table 3. It is envisioned that there may be a need for multiple GOFIRS to “patrol” various 
segments of the Space Station in a relatively slow “inspection” mode. It is also envisioned that 
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one GOFIRS may be assigned to each of the outboard rotating solar wings while one or two more 
units will patrol the fixed central portion of Station. 

This would circumvent the need to devise a means for crossing the alpha rotary joints. For 
example, 4 such GOFIRS dispersed in this manner could make a complete inspection of the 
Station’s primary structure in approximately 60 hours at an inspection rate of 2 feet/minute. 

Another point to be made is that a “low level” GOFIRS type system utilizing current state- 
of-the-art robotics technology could be developed in time for a shuttle flight demonstration prior 
to launching the Space Station. 

5. Description of Capabilities 

A conceptual illustration of a “wheel driven” GOFIRS appears in Figure S. A description of 
features and capabilities is summarized in Table 4. On board microprocessor capability will be 
needed to perform the inspection/repair functions, control the GOFIRS motion, and to perform 
in situ diagnostic analysis of sensor data. This data can be logged and telemetried back to the 
crew capsule or ground using data compression techniques at some later time or at once if an 
emergency requires immediate attention. Location of the defect and information for guidance 
could be obtained by encoded magnetic strips like “bar codes” circumscribing the struts. Optical 
sensors could be used in place of these magnetic strips. 


cameras 



Figure 5. Global Operation Flight 
Inspection Repair System 


Table 4. Capabilities 
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Teleoperator or telerobotic repair would be accomplished with one or two dexterous manip- 
ulators having the appropriate tactile/force feedback sensors and utilizing multiple cameras 
(stereo-vision). Tools' and replacement parts would be carried to facilitate either crew EVA or 
telerobotic On-orbit Replaceable Unit (ORU) changeout. 

The platform would be motorized being either a wheel driven or propelled by a coilable mast 
arrangement to be discussed later. In the wheel driven variant, two “steerable” and “latchable” 
drive wheel bogies (See Figure 6) would be lightly spring loaded against the tubular strut with soft 
polymer coated wheels. The allowable contact pressures to prevent damage to the struts would be 
determined by extensive tests. However, in-house tests of a protype, aluminum-clad, carbon- 
graphite epoxy tube of the required size sustained a point load of over 200 pounds without 
damage. Anticipated wheel loads for the GOFIRS would be at least one order of magnitude lower 
than this. A tube clamp mechanism with a large footprint could be incorporated if needed to react 
large torques during pan removal and replacement. 

The large MRMS/Transporter will make use of pins attached to the truss nodal connectors 
to crawl along the station. These same pins, as shown in Figure 7 could be used to pivot the 
GOFIRS from strut to strut. In one arrangement, shown in Figure 8, a simple jaw type grip, 
equipped with a gear drive could swing or pivot the platform about the pin. The steerable wheel 
bogie (Figure 6) can “spiral” the platform to a sideface (see Figure 7) and the pivoting action can 
then take place. Diagonal members (not shown) can be reached by pivoting 45 degrees. Pivoting 
180 degrees will permit continued motion along the same longeron. Thus through various combi- 
nations of pivoting and spiraling virtually any strut member can be reached without additional 
tracks or alterations to the primary structure. 
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Rechargeable batteries would provide energy for locomotion, microprocessing and robotics 
functions. A power bus outlet could recharge the batteries after a predetermined tour. 



Figure 6. Wheel Bogie Figure 7. Pivot Sequence 

Additional capabilities such as detecting, replacing and/or cleaning solar array panel seg- 
ments could be achieved with GOFIRS as illustrated in Figure 9. Here the GOFIRS shuttles back 
and forth on a tubular strut supported by two expandable masts. Scanning across the array, 
“window washer fashion”, the infrared sensors on board the manipulator are mapping tempera- 
tures to isolate malfunctioning solar cells. 


In the event that the tubular struts themselves cannot be used for support, a twin mast driven 
platform is envisioned, as illustrated in Figure 10. Appropriately expanding and contracting the 
masts will provide linear motion. These coilable masts are similar to those conventionally used to 
deploy flexible solar arrays. A turn-table bearing will permit the platform to assume any planar 
orientation. The pivoting function will occur in the same manner as before with the exception 
that the forward mast segment will translate and fold (see Figure 10) to allow motion along the 
side face. Stiffness and buckling strength of the twin masts are not anticipated to be a problem. 
Adequate cycle life of the flexible battens must be established. 



Figure 8. Nodal Pin Figure 9. Solar Array 

6. Telerobotics 

In another variant, as shown in Figure 1 1, the GOFIRS serves as a combined crew transport 
and man-controlled servicer. Here the astronaut has a direct visual link with the repair or chan- 
geout activity while crew members aboard the Station or on the ground can participate if need be. 
The system is equipped with one or possibly two dexterous manipulators to facilitate teleoperator 
repairs. 
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Figure 10. Twin Mast Propelled Platform 


Figure 12 illustrates a more advanced telerobotic service configuration for GOFIRS. In prin- 
ciple, the robotic unit pictured here can be the same as that developed for the MRMS or Robotic 
OMV. The addition of the robot strengthens the GOFIRS capability in performing repairs and 
making replacements but will undoubtedly add to the cost and size of the platform. A cost-to- 
benefit assessment of the degree of robotic sophistication will be needed. 



Figure 11. Crew-Controlled Servicer and Transport 
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Figure 12. Flight Telerobotic Servicer Mounted Platform 

7. A & R Technology For Space 

Substantial progress has been made in the development and application of automation and 
robotic technologies for ground based applications. Industrial robots are now commonplace in 
factories. However, the repetitive, well structured, highly defined tasks that shop robots are well 
suited to perform are not commonplace in space. In space applications the tasks are often very 
diverse, less frequent and highly complex. 



Figure 13. The A <£ R Challenge! Figure 14. Enabling Technologies 

Figure 13 best illustrates this dichotomy of needs. On one hand, robots are ideally suited to 
autonomously perform well-structured tasks. On the other hand, teleoperated manipulators, 
such as those used in the nuclear industry, can be adapted to perform less structured tasks due to 
their human operators. However, the price for this adaptability is the expenditure of dedicated 
operator time, a precious commodity aboard the Space Station. 

The “challenge” is to bridge the gap between the adaptability afforded by a teleoperator and 
the autonomy offered by a robot. Some see this bridge as a “tele-robot”, one system offering 
both capabilities, while leaning toward more teleoperation in the early years. 
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The GOFIRS concept presented here enbodies this duplicity of capabilities. Moreover, it 
offers the opportunity to have a distributed robotic capability about the Space Station, in the 
same manner as robotic machines are distributed about our factories. Consider the effectiveness 
and reliability of several smaller machines, simultaneously performing a sequence of simpler 
tasks in comparison to one super-sophisticated machine required to alone perform the cumulative 
tasks of the team of smaller robots. 

Despite significant on-going progress in many areas of A & R technology, our current level 
of technology would only support a space based telerobot having relatively low level capabilities. 
A partial, by no means complete, list of areas where strengthening is warranted appears in Figure 
14. The work needed in the robotics area can most quickly be envisioned by the somewhat face- 
tious notion of adapting a 2-ton shop robot to become a space-qualified, flight manipulator. One 
that has sufficient dexterity to remove a defective circuit board from a delicate instrument if 
required. Reliabilities associated with today’s industrial robots are far from those required for 
precision space mechanisms. Few, if any, have been designed to operate in a vacuum. Unfortu- 
nately, the effort needed to develop and demonstrate the relevant electro-mechanical technologies 
for a space-worthy, multidegree-of-freedom robot is sometimes under-appreciated. 

Other areas requiring continued attention include a range of sensor and detector technolo- 
gies, with high emphasis on vision related systems for teleoperation (see Figure 14). A whole 
family of inter-related activities fall under the area of machine intelligence, including task plan- 
ning and reasoning, control execution, human interfaces and system architecture. The ability to 
make in-situ, real time, autonomous assessment of sensory inputs will be particularly important 
in enhancing crew productivity. 


8. Conclusion 

A concept for a self-intelligent, mobile platform is presented which can perform many of the 
inspection and maintenance activities envisioned for Space Station. Routine inspection related 
tasks can represent the single greatest expenditure of crew time given the shear size and complex- 
ity of Station’s support systems. Several sensor-equipped, mobile platforms or GOFIRS working 
together with health monitoring sensors internal to these subsystems would be of great value in 
identifying not only the location but, moreover, the extent, hence urgency, of the defect. In this 
way, GOFIRS performs as a “scout” for the crew and relieves scheduling of the large teleopera- 
ted MRMS. 

Features of the concept include the ability to move about Station without the need for special 
tracks or cables. Virtually all exterior and interior areas within the Station’s framework are 
accessible. 

The GOFIRS concept is modular, accomodating more advanced robotic capabilities as they 
evolve. In its simplest form, GOFIRS is an inspection cart with some crew and tool transport 
capabilities. Rudimentary teleoperation capability can be added with the addition of a flexible 
manipulator and vision equipment. Later, a more advanced flight telerobotic servicer unit could 
be accommodated . In this way a stepping stone approach can be taken . The advantage is the ability 
to demonstrate the concept at some low level with technologies available today at minimal risk. 

The next phase of the work is to establish the performance requirements of a GOFIRS type 
system in relation to operational inspection and servicing activities to be scheduled on board the 
Station. This will set the frame work of preliminary design to arrive at the balance between cost, 
risk and capability. Suitability for early flight demonstration would also be assessed. 
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1. Abstract 

Research and development projects have characteristically followed development processes 
structured around well-defined, but loosely organized, research goals. This particular 

approach differs from the standard, application-specific, product development found in the 
private sector. Nevertheless, research and development often follows a less defined 
application route because of the substantial amount of technical risk associated with its 
research goals. Novel system engineering techniques have been developed and applied to 
establishing structured design and performance objectives for the Telerobotics Testbed that 
reduce technical risk while still allowing the testbed to demonstrate an advancement in 
state-of-the-art robotic technologies. To establish the appropriate tradeoff structure and 
balance of technology performance against technical risk, an analytical data base was 
developed which drew on 1) automation/robot-technology availability projections, 2) typical 
or potential application mission task sets, 3) performance simulations, 4) project schedule 
constraints, and 5) project funding constraints. Design tradeoffs and configuration/ 
performance iterations were conducted by comparing feasible technology/task set configurations 
against schedule/budget constraints as well as original program target technology objectives. 
The final system configuration, task set, and technology set reflected a balanced advancement 
in state-of-the-art robotic technologies. while meeting programmatic objectives and 
schedule/cost constraints. 

2. Introduction 

Funding limitations in both pi ivate and government sectors often make it difficult for 
research and development environments to operate totally independently of mainstream 
applications of potential products resulting from the research. Similarly, the Telerobotics 
Testbed, a research and development effort, is being viewed as a soucce of advanced seed 
robotic technology for the Space Station Flight Telerobotic Servicer (FTS) . The near horizon 
for first-element launch (FEL) and initial operational capability ( luC) (t.e., the early to 
mid-1990's) places some pressure on the testbed breadboard etfoit to tailor its technology 
thrusts, and potential applications, towards these near-term developments. One of the 
challenges associated with defining the testbed breadboard development program is finding the 
appropriate balance between establishing an aggressive technology development program, yet 
maintaining a viable application channel with the Space Station FTS environment and 
development schedule. From an operations research viewpoint, this situation represents the 
classical problem of satisfying several competing objectives with limited resources. 
Although it would appear that classical linear programming or "branch and bound* optimization 
techniques could *ie applied as solution structures to the competing objectives problem, in 
fact the introduction of key intangible (t.e.. not readily quantifiable) variables made the 
solution of the problem not wrvnediately amenable to a rigorous mathematical representation. 
Nevertheless, optimization techniques such as branch and bound provided a structure for 
obtaining progressive, feasible sets of solutions that could be independently examined until 
a "reasonable" solution to .he performance versus technical risk tradeoff problem was found. 
The following paragraphs discuss 1) how the ovetall problem and solution structure was 
developed, 2) the tradeoff variables (both tangible and intangible), 3) the rationale behind 
the derivation of feasible solution sets, 4) the selected feasible solution and associated 
bounds, and 5) supporting data. 

3. Problem Definition and Solution Structure 

The first step in obtaining a solution to the competing objectives problem was to 
establish a concise definition of the objectives and constraints. The major variables that 
’needed to be satisfied in the tradeoff process were as follows: 

1. Programmatic technology objectives - Addresses the overall approved technology goals 
jointly agreed to by the research sponsor (NASA Office of Aeronautics and Space 
Technology lOAST]) and the responsible research or gan i zat ions (Jet Propulsion 
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Laboratory, Langley Research Center, Marshall Space F light Center, Johnson Space 
Center, Ames Research Center, Goddard Space Flight Center, and Massachusetts Institute 
of Technology.) 

2. Viable mission task set - Refers to the development of an application environment 
which is both feasible (in terns of technology performance capabilities and 
constraints) and representative of a real-world use of the technology. 

3. Schedule - Addresses the time constraint associated with completing the technology 
objectives as part of normal progr a mm a tic planning/assessment, and meeting other 
outside schedule needs such as the FTS FEL/IOC development and qualification 
milestones. 

4. Cost - Refers to the budgetary constraint imposed at the programmatic control 
organization (NASA OAST). 

5. Performance - Addresses the capability of the hardware and software to actually 
execute and successfully complete a selected task set (a measure of technical risk). 

6. Technology availability - Refers to the actual state of maturity of a given technology 
element as measured against state-of-the-art and in the context of the overall system 
capability to perform a selected task set. 

In examining each of the above variables in terms of "objectives* and "constraints" it 
became clear that the first two variables (technology objectives and mission task set) 
represented the primary optimization objectives. The ability of the program, and actual 
system design, to reach these objectives would be subject to the constraints imposed 
respectively by schedule, cost, hardware and software performance limitations, and the 
relative states of achievable maturity of the component technologies. Mathematically, the 
optimization problem could be stated as follows: 
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The above formulation basically states that it is desirable to maximize the overall 
targeted technology capability (T) and feasible application task performance capability (a) 
subject to 1) the respective technology development schedules (s) not exceeding the overall 
programmatic schedule (S), 2) the respective technology development costs (c) not exceeding 
the overall programmatic cost ceiling (C) , 3) the respective technology performance 
limitations (p) being commensurate with the overall progr aramatic technology performance 
objectives (P), and 4) the aggregate achievable technology maturities (t # ) being greater than 
the overall state-of-the-art technology level (T’). The above formulation serves the purpose 
of providing a clea statement of the competing objectives problem. However, from « 
practical standpoint it is very difficult to actually measure all of the above variables. 
For example, the technology objectives and application task set do not lend themselves to 
quantification in the same sense as cost and schedule. Similarly, setting the 
state-of-the-art technology baseline and comparing the composite testbed technology maturity 
level against that baseline is also difficult to quantify. Therefore, these three variables 
represented important, but intangible variables. The remaining variables (schedule, cost, 
and performance) represented the tangible variables. • 


In order to cope with the intangible variables, a more empirical approach was taken to 
structuring the optimization problem. Keeping the objective function and constraints the 
same, a modified branch and bound technique was formulated that provided a tradeoff structure 
that could accommodate both quantitative and qualitative representations of the objective and 
constraint vaiiables. 
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Therefore, the next step in formulating the solution was to tailor the branch and bound 
optimization structure to handle both qualitative^ and quantitative decision data. By 
definition, the branch and bound optimisation technique starts by setting a bound on the 
objective function (Ref. 1). Rest, the technique requires that the set of all feasible 
solutions (t.e., in this case the technology and application task sets) first be partitioned 
into several subsets. Because the objective of the exercise is to maximise the chances of 
meeting the original technology objectives while exercising those technologies in the most 
robust application environment possible, any subset of alternative technologies and 
applications that does not meet the original objectives is eliminated. Each subset is 
evaluated against the objective function and constraints until a solution is found that meets 
all conditions. In the absence of a clear-cut analytical solution to the competing 
objectives problem, a decision network was designed that allowed the subset partitioning and 
evaluation steps to be completed in exactly the same spirit of the branch and bound solution 
structure outlined earlier. This decision network is shown in Figure 1. 


START 



Figure 1. Feasible Solution Decision Network 

Figure 1 displays the serial decision process that allows the subsets of feasible 
solutions to be filtered out of a large group of candidate technologies and applications. 
Note that the decision structure is designed to be an "and" decision gate so that both 
objectives and constraints nus. be simultaneously satisfied (as implied in eqs. 1-5) to 
obtain a "reasonable" solution. It should also be noted that although the above structure 
provides a reasonable solution, by design, it does not yield the rigorous, analytical 
numerical solution that linear programming or classical branch and bound optimization 
techniques yield. 

4. Data Base 

The above objective and constraint variables were supported by an extensive quantitative 
and qualitative data base. These various data bases are summarized below: 

1. Programmatic technology objectives (qualitative) - The programmatic objectives were 
established at the onset of the testbed project (Ref. 2). The overall Phase 1 (FY 
1987/1988) program objectives were 1) automated object acquisition and tracking. 2) 
video-based locat ion/or ientat ion of simple objects. 3) off-line coordinat ion- level 
telerobot activity planning, 4) an architecture for coordinated planning/diagnostics 
for telerobot command and control, 5) dual-arm coordinated control with hybrid 
force/torque, position, and rate feedoack, 6) dual force reflecting hand controllers, 
stereo display, and fused force/torque video feedback for teleoperation, 7) an 
architecture for run-time control of the telerobot with the capability to interpret 
and execute task primitive commands generated by the act:*. ; ty planner, and 8) a 
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distributed, suit i -processor command and control hierarchy with the capability to be 
Modular ly upgraded and provide simple error recovery. 

2. Viable Mission task set (qualitative) * A fairly extensive literature search was 
conducted to establish an application task set in which to develop and test the 
various technologies and overall telerobot syste* (Refs. 3-13). At the onset of this 
portion of the analysis it was assumed that the roost viable application of the 
telerobot, in the near term (per the FTS augroent to extravehicular activity), would be 
for on-orbit assembly and servicing. Therefore, the application task set was sought 
priroarily in planned, or historical, on-orbit servicing activities. Sky lab and 
Shuttle historical experiences were roost useful. Unfortunately, proposed Space 
Station-related servicing missions such as Space Telescope were not defined to a level 
of detail that would facilitate an accurate mapping between servicing functions and 
needed technologies. Ultimately, the Solar Max repair mission provided a full array 
of detailed servicing tasks that was sufficiently gianular and representative of 
probable FTS servicing activities so as to provide a good starting application subset. 

3. Schedule (quantitative) - The schedule constraints imposed on the project were 1) a 
demonstration of core technology elements by end of FY 1987, 2) followed by a full 
integrated demonstrat ion of the complete telerobotic breadboard system by end of 
FY 1988. 

4. Cost (quantitative) - The cost constraint for the project for the three-year effort 
starting FY 1986 (including funding outside leverage from other NASA centers, 
industry, and universities) was projected to be approximately $20M. 

5. Performance (quantitative) - The performance envelope of the technologies was derived 
from the actual physical capabilities and constraints of the hardware and software 
used in the research laboratory. For example, the vision subsystem was able to 
provide fixture location to within 1 mm and resolve unoccluded fixtures (within the 
constraints of the internal object model software) such as small panels, handles, or 
bolt heads. The PUMA 560 arms (typical of nationwide laboratory hardware) used in the 
control technology development, had specified reach envelopes, joint movement 
constraints, and load-handling capabilities. Once a task set, object library, and 
task data base (object locations, forces, torques, etc.) were established, the system 
performance was simulated on an IRIS dynamic computer display system to obtain a rough 
estimate of system and application feasibility. A single frame of the dual arm 
servicing simulation is shown in Figure 2. 



Figure 2. Dual Arm Telerobot Servicing Simulation (IRIS) 

6. Technology maturity and availability Uual i tat ive) - when faced with hard schedule and 
budgetary constraints, projects must set their sights on technology goals which 
r*pr«sent both an advancement as well as 9 realistic, achievable objective. Although 
some studies have been done which suggest both maturity levels and time frames for the 
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breadboard and fully operational versions of advanced automation technologies (see 
Refs. 14, IS, and 16), generally it is extremely difficult to bound, or constrain, a 
qualitative variable using an upper bound which has a fairly large variance itself. 
This problem is compounded when considering other constraints such as setting 
technology goals that enable the breadboard development (i.e., the FY 1988 schedule 
constraint) to transfer technology in a timely manner to both the FTS brassboard and 
fully operational configurations. This development constraint implies that a distinct 
time frame is needed to move through all the development stages as shown in Figure 3. 
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YEARS TO FULL OPERATIONAL CAPABILITY FOCI 

SIMPLE SYSTEMS 'MPLEMENTATION OF A SINGULAR ACTION; OPERATIONS 
GENERALLY INDEPENDENT of OTHER FUNCTIONS UNIQUE APPLICATIONS ALTHOUGH 
BASIC PR NCIPLES WELL UNDERSTOOD 

MODERATELY COMPLEX SYSTEMS MULTIPLE INTERACTING FUNCTIONS OR ACTIONS 
COMPLEX CONTROL LOGIC OR NETWORKS BASIC IMPLEMENTATION TECHNIQUES 
SIMILAR TO PREVIOUSLY DEVELOPED SYSTEMS 

complex systems multiple interacting functions or actions complex 

CONTROL LOGIC OR NETWORKS REDUCTION TO PRACTICE OF DESIGN CONCEPTS 
WHEN COMPARABLE SYSTEM HAS NOT BEEN DEVELOPED 


Fiquce 3. Technology Development as a Function of Readiness Levels and Time 


Therefore. rather tnan establishing an upper constraint for the maturity variable, a 
sta te-of -the-ai t baseline was established and used as a Known, lower bound. The state-of-the 
art Lower bound then simply had to be exceeded while simultaneously providing a viable 
bieadboard cuiif i gu i at i on that could appi opi i a t e l y meet FTS schedule constraints. The 
state-of-the-art baseline was set against available, working engineering mode 1 s and included 
I) s ensing and p ercep tion - simple labeled and unlabeled obiect tracking with manual 
acquisition; 2) task planning/ reasoning - off-line sequence generation and no we l 1 -st r uctu r ed 

human-robot cooperative plan generation; 3) oper at or inter f a ce - dual arm t e leoper at i on . 

limited real-time computer graphic displays, stereo vision, limited external state sensing, 
limited oper at or /wot kst at ion integration. no traded control between teleoperation and 
autonomous states; 4) contr ol e xecu tion - mode l -based single arm control or teach pendant, 
leader - fo i lower dual arm position control, limited hybrid control. 5) control architecture 
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end infarction - limited hierarchic*! control* centralized process i ng/Maory f coordination 
love! control in structured manufacturing environments* distributed processing architectures, 
teleoperation and autonomous control not traded, limited hierarchical error management. 

5. Tradeoff Results 

The last step in the analysis was to execute and re-execute the Figure 1 decision 
structure until a reasonable solution mas obtained which met both the objective function and 
constraints. The iteration process commenced with publishing an application tash set 
(drawing on the full Solar Max servicing scenario) along with the projected commensurate 
implementation technologies. Immediate problems were encountered because 1) the real-time 
reconfiguration tash elements associated with main electronics box (MEB) exceeded the tash 
planning capability of the system. 2) object masses and electric socket removal forces 
exceeded the load characteristics of the PUMA arms. 3) some component disassembly sequences 
exceeded the hardware and software control characteristics of the PUMA arms and control 
algorithms, and 4) the large array of geometric shapes associated with the servicing 
environment exceeded the vision system CAD data base. The servicing scenario was downscaled. 
The task-related objects were redesigned to accommodate the PUMA constraints and simplified. 

In the manner described above, each application tash set and corresponding technologies 
were reviewed with the various subsystem research engineers against schedule constraints, 
budgetary limitations, hardware/software limitations, and the state-of-the-art baseline until 
a subset of each was obtained which satisfied all the objectives and the constraints. The 
corresponding solution set is shown in Table 1 (Ref. 17). 


Table 1. Telerobot Application and Technology Solution Subsets 


Application Tash Set 


Technology 


1. Capture/dock slowly rotating 
satellite (1 rpm) 

2. Verify initial object in 
task sequence (MACS) 

3. Remove star tracker covers 
on MACS 

4. Confirm auto sequence plan 


5. Teleop traded off to auto, 
verify/grasp bolt wrench 


6. Remove MACS retaining bolts 


7. Remove/ rep lace MACS 


8. Auto traded off to teleop 
for satellite repositioning 


9. Remove MEB thermal blanket 

10. Teleop traded off to auto, 
hinged panel door opened, 
simplified MEB electrical 
connectors removed. MEB 
removed and replaced 


Automated labeled object 
acquisition, tracking, dual arm 
servoing 

Automated stationary object 
verification 

Teleoperation under alignment/ 
accuracy/force constraints 
(dual arm) 

Operator-AI planner interaction 
(operator can update object 
location, confirm plan, or 
update a task monitoring point) 
Automated object verification, 
plan execution, hierarchical 
control with limited error 
recovery 

Automated object verification, 
hybrid force/position and force/ 
torque control with trimming 
Automated object verification, 
dual coordinated master/slave 
arm control, simple collision 
avoidance, position and rate 
control 

Dual arm teleoperation, 
position/alignment control 
(video, stereo. 6 DOF hand 
control, and voice camera 
control) 

Same as 6 above, handling 
flexible objects 
Same as 4 through 7 above, 
limited automated flexible 
object handling, precise auto- 
mated control in simple obstacle 
field with quarded motion along 
an arc 


The above table is somewhat abbreviated for summary purposes. However, the complete 
detailed application task set and technology correlation is provided in the Telerobot Testbed 
functional requirements (Ref. 17). By far. the largest improvements in the respective 
technologies over state-of-the-art revolved around the vision-based fixture update and its 
integration with the control execution, the integration of the planner with the control 
execution, the auto to teleop traded control, the dual arm coordinated control, and the 
distributed control hierarchical design with on-line (although simple) error management woven 
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throughout the hierarchy. The application task set* although simplified to meet performance 
and technology constraints* still provided a viable environment reasonably close to projected 
orbital replacement unit (ORU) removal /replacement FTS tasks. Finally* the selected 
technology subset was reasonably in-line with schedule/cost constraints; and* although 
composed of both state-of-the-art technologies and evolutionary (as opposed to revolutionary) 
improvements over other state-of-the-art technologies* the selected subset appeared 
achievable in a manner commensurate with supporting the out-year FTS development. 

6. Conclusions 

The revised branch and bound solution structure augmented with the supporting data bases 
and system simulation provided an excellent blueprint for obtaining a reasonable solution to 
an extremely difficult tradeoff problem. This technique has proven very useful for 
structuring the Telerobot Testbed research and development program to be sensitive to 
real-world demands and constraints. The technique is presently being employed to start 
negotiating and planning the 1990 demonstration. 
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1. ABSTRACT 


/ 


t 


p presents the concept of a aeries of dedicated robotics manipulators that would be 
resident in the subsystems of the Space Station/ These would be used to do Orbital 
Replacement Unit (ORU) exchanges, inspection of the components, and in certain cases subsystem 
assembly. By performing these well definded tasks automatically, higher crew productivity 
would be achieved. In order to utilize the robots effectively ORD's must be designed to allow 
remote release and quick disconnection of the electrical, fluid, and thermal connections. The 
robot must be of a modular design for ease of maintenance and must have an adaptive control 
capibility to make-up for slight errors in programming. 


2. INTRODUCTION 

The construction, operation and maintenance of the Space Station will present many 
challenges. In the past space based systems required that the components be certified for 
the life of the mission with little or no opportunity for servicing. Since the Space Station 
will be a permanent manned platform in space the opportunity exists to not only service and 
maintain the components but also to update them as improved technology is developed. With 
this in mind it is important to utilize available resources and techniques to design 
the station to be as easily assembled, /serviced, and maintained as possible. In addition 
it is important to keep in mind that/the purpose of the astronauts presence on the space 
station is to provide support for the/ experiments and manufacturing efforts in space and 
not to be incumbered with the mundane tasks of station maintenance and servicing. 

3. METHODS FOR ASSEMBLY, SERVICING, OPERATION, AND 
MAINTENANCE 

Three methods exist for assembly, servicing and maintenance of the Space Station. These are 
Extra-Vehicular-Activity (EVA) , Inter-vehicular Activity (IVA) using remote teleoperated 
manipulators and automation in the form of fully automatic robotic manipulators. 

EVA 

EVA provides the greatest flexibility of the three methods since the astronaut can 
interact directly with the system. However, this presents the highest' risk to the 
astronaut. In addition several other drawbacks exist. This method is expensive in that the 
life support systems (EVA suit) are expensive to maintain with costs estimated to be $80,000 
per EVA hour. Also when comparing the time to perform tasks during EVA to normal activities 
on the ground it can take up to eight times as long to perform the same task. Additional 
time must be spent in pre and post EVA activities. With these facts in mind it would 
thus be important for the astronaut to remain in the station unless absolutely required to 
leave . 

IVA 

IVA allows the astronaut to remain in the station and perform tasks outside using a remote 
teleoperated manipulator. Presently two systems the Flight Telerobitic Servicer (FTS) and 
the Mobile Service Center (MSC) have been identified to do this. Teleoperation allows a high 
degree cf flexibility in positioning the manipulator to perform the tasks since it is under 
continuous control of the astronaut. However, it is not a simple task to position the 
manipulator even using multiple cameras and displays. This method is also time consuming and 
can take up to sixteen times as long normal ground activities. In addition the astronauts 
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attention is required even for the slap lest activities such a moving along the truss. Also 
the manipulator is making unplanned moves that can impart inertial loads on the station 
resulting in vibrations or effecting the validity of low g experiments. 

ROBOTICS 

The last method is to use fully automated devices such as robots. Robotic devices can be 
preprogrammed to perform tasks that would not require the direct attention of the astronaut. 
As part of the program the robot could pause at critical points and the allow the astronaut to 
view the operation and correct the motions, if necessary, before continuing. The ability to 
control the robot remotely would be provided as a back-up. Because of the relative 
inflexibility of the robotic devices the tasks would have to be well defined. These tasks 
would include 080 removal and replacement, component inspection, and, in limited cases, 
assembly. This would free up the astronaut to do other less defined tasks such as shuttle 
unloading and satellite capture and servicing using the teleoperated devices. It would be 
difficult to program a single mobile robot to service all parts of the station. A more viable 
solution would be to provide dedicated robotic devices as an integral part of the Space 
Station subsystems. 

4. DEDICATED ROBOTICS 

This concept provides for modular robotic devices that would be dedicated to the assembly, 
operation and servicing of particular subsystems on the Space Station. By limiting the tasks 
required of a particular robot to those for that subsystem the complexity can be 
significantly reduced. 

ORU REPLACEMENT 

The primary task of the dedicated robots would consist of removal and replacement of ORUs. 
The replacement would be performed on a preprogrammed basis and would not require intervention 
by the astronaut. Because the robot is local to the subsystem, it could aid in the diagnosis 
of the failure through subsystem testing. If the immediate cause of the failure cannot be 
identified, or narrowed to a particular # ORU the robot would be available to do ORU swapping to 
determine which component had failed. 

ASSEMBLY 

Well defined assembly tasks such as first time insertion of ORU's into the subsystems could be 
accomplished using the robots. With proper considerations given component design such as a 
common interface and methods of locking the component into place on the structure the robot 
could perform more complicated assembly operations. These same techniques would be useful in 
reducing the complexity and time required to do assembly using EVA or IVA. Using multiple 
robots in the individual subsystems would allow simultaneous operations to proceed. 

INSPECTION 

If it is determined that the cause of the failure is external to the ORU, the astronaut could 
use the robots sensors and vision system to help identify the cause of the failure. The 
astronaut would either program the robot to make the repair or use the information obtained to 
help plan an EVA. Preventative inspections and service of the components in the subsystem 
would help to predict and prevent catastrophic failures. These could be carried out on a 
regular basis without astronaut attention and be reported directly to the health monitoring 
system. 

PROGRAMMING 

A primary concern for use of robots would be the programming of the many and various tasks. 
This could be accomplished using graphic simulation and Offline Programming (OLP) based on a 
CAD data base of the Space Station. The majority of tasks can be identified, programmed, and 
simulated prior to launch. These programs would be stored and executed as required during 
operation. However, many tasks would require programming during flight. This could be 
accomplished by ground crews using the OLP and simulation stations, and then uploaded to the 
station for execution. The astronaut could perform the simulation (and programming if 
necessary) on board the Space Station to verify that the task will be accomplished to his/her 
satisfaction. The OLP/ simulation system would be provided with a user friendly interface. 
Specifying the particular subsystem in question would bring a simulated cell onto the display 
with the robot and all components. By simply indicating the positions to move to or the task 
co perform the program would be simulated and down loaded to the robot for execution. A 
similar system to this is being developed at Rocketdyne for welding the Space Shuttle Main 
Engines as shown in figure 1. 


76 



FIGURE 1 



END EFFECTORS 

Critical to the operation of the robot, or any remote manipulator would be the ability to 
interface with the various ORUs. The end effector would be designed to mate with the ORU as 
shown in figure 2 and include a mechanism for actuating the built in locking and ejection 
system. This will require the ability to easily change end effectors to accommodate various 
ORUs within the subsystem. In addition a compact end effector which houses multiple sensors 
could be provided. This would be used during the regular inspection periods and for trouble 
shooting and diagnosing problems. 


FIGURE 2 

TASK SPECIFIC END-EFFECTORS 
REMOVE AND REPLACE ORU’S 



SENSORS 

Offline programming and graphic simulation provides a path for the robot which will avoid 
collisions. To make up for the variation between the programmed path and the actual path 
both vision and tactile sensors will be required. The vision system as shown in figure 2 will 
allow the robot to adapt to variation in the location of the ORU interface and position the 
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robot for final docking with tha ORU. Additionally tha system would includa an optical 
charactar raadar to idantify tha ORU. A forca/torqua aanaor in tha vriat would ba usad to 
adapt lvaly position tha robot to prsvant jamming and provlda a smooth, parallal ins art ion. 
This is also shown in figura 2. During tha ins pact ion task, various typas of sansora will ba 
required. Voltage, currant, logic, and communication chacks can ba performed with a •plug in" 
typa connactlon to ports on tha ORUs. Naasurasant of aachanlcal propart laa such as vibration, 
temperature, waar, torqua and surfaca da facts ara sora difficult. Figura 3 shows a concapt 
for a compact and affactor with multipla sansors that ara flbsroptically couplad to tha 
control alactronlcs. 


FIGURE 3 

END EFFECTOR FOR COMPREHENSIVE INSPECTION BY 
DEDICATED SERVICING ROBOTS 



* ! 

5. ROBOT DESIGN CONSIDERATIONS 

By limiting tha anvalop to a particular subsystam on tha station tha robot manipulator can ba 
dasignad as a rigid structura thus simplifying tha control. A modular design would allow 
similar components to be arranged in different configurations to accommodate tha variations in 
tha tasks. Due to thn lack of gravity tha robot need only provlda tha fores necessary to 
accelerate and decelerate the ORU. By programming very low accelerations and decelerations tfe 
axis motors and drives can be made small and compact. This would have the additional advantage 
of reducing tha inertial reaction on the Space Station structure. The use of composites in the 
design of the manipulator links can provide a light weight and rigid manipulator which is 
capable of moving large masses. Also use of direct drive motors would reduce tha weight. 
Figure 4 shows the design of redundant 7 axis robot with a 90 inch modular arm whose total 
weight with control is estimated to be 46 lbs. 

FIGURE 4 

ROBOTIC SYSTEM WEIGHTS 
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6. ADVANTAGES TO DEDICATED ROBOTS 

Thar* art several advantages to the use of dedicated robotics over other methods discussed. 
Present plans call for the use of two teleoperated devices, the Mobile Service Center (MSC) 
and the Flight Telerobotic Servicer (FTS) , to perform all remote assembly, service, and 
maintenance tasks on the Space Station. This would create a problem if more than two tasks 
were required at the same time especially if the tasks were on opposite ends of the station. 
Dedicated robots could perform simultaneous tasks on various parts of the station in many 
cases vith-out requiring the direct attention of the astronaut. 

REDUCED ORU COUNT 

Many critical systems on the Space Station will require double, triple or quadruple 
redundancy. By utilizing dedicated robots failed components can be replaced immediately 
rather than waiting for a planned service interval. This would alleviate the need to provide 
as high a redundancy level as predicted and thus reduce the ORU count and number of spares 
required. 

PREVENTATIVE MAINTENANCE 

By providing the local ability to do inspections and subsystems checks with the robot, it will 
be easier to determine the cause of the failure and to identify the failed component. Also by 
performing regular inspections with the robot, failures can be predicted and corrective action 
taken before a catastrophic failure occurs which could damage adjacent components. 

DESIGN FOR SERVICE 

As mentioned before, space based systems in the past required that the components be certified 
for the life of the mission. These former systems in comparison to the Space Station were 
relatively short lived and less complex. Since the mission life of the Space Station is 30 
years, components are required to have a mean time between failure (MTBF) of from 10 to 30 
years. In order to obtain these high MTBF ' s, significant development and manufacturing costs 
will be incurred. With the use of dedicated robots the ability to maintain and service the 
Space Station would be significantly enhanced. This would provide the opportunity to design 
the components for a shorter service life of fro* 1 to 5 years and thus avoid some of the 
initial development and fabrication costs associated with commissioning of the Space Station, 
in addition, as new technology is developed obsolete components could be easily replaced so 
that the Space Station remained at the highest state of the art obtainable. 

7. REQUIREMENTS FOR USE OF DEDICATED ROBOTS. 

In order to utilize robots consideration will have to be given to the design of the Space 
Station and its subsystems and components. These same considerations will also provide for 
ease of service and assembly by EVA and IVA. 

ORU DESIGN 

The Orbital Replacement Units should be of a modular design and provide for a common interface 
between the ORU and the manipulator end effector. A range of interface sizes should be 
provided to accommodate the different size ORU's. This interface should be designed with 
adequate lead-in so that a slight misalignment of the end effector would not cause jamming. 
An alignment target should be provided so that the vision system can locate and do final 
positioning for connecting to the ORU. In addition identifying markings should be provided 
adjacent to the target so that they may be verified by the optical character reader. 

Quick disconnects should be provided for electrical, communication, fluid and thermal con- 
nections to the ORU. Fluid connections should contain a check valve shut off and a leak 
detection device with double seal arrangement to determine if the check valve has sealed. The 
robot can be programmed to pause for a leak check, if a seal has not been achieved the removal 
can be aborted. 

A method to connect/lock and to unlock/eject the ORU should be provided so the manipulator 
is not required to push or pull on the ORU. This will prevent uncontrolled motions by the 
manipulator when the ORU is removed and provide the forces necessary to overcome the required 
contact pressures. The actuator for this mechanism would be contained in the end effector of 
the robot. 

Many of the initial start-up and servicing problems on complicated systems such as the Space 
Station are associated with the connections to the individual components. An additional 
requirement should be to route the electrical cables, fluid lines and connectors in such a 
manner that they may be easily inspected and repaired remotely from the front panel. A concept 
of this ORU design is shorn in figure 5. 
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FIGURE 5 


ORU SERVICEABLE CONNECTIONS 



CAD DATA BASE 

In order to provide for offline progressing and graphic simulation of the robot tasks an 
accurate CAD data base of the Space Station will be required. This will assure that the robot 
path vill not interfere with other portions of the station and that the actual robot sot ions 
can be executed. In addition this data base vill be invaluable in configuration control and 
redesign during growth of the Space Station. 

DELIVERY OF ORUS 

Replacement ORU's, components, and end effectors must be delivered from storage to the 
individual robot systes. Failed cosponents and unused end effectors must be returned for 
storage or repair or delivery to earth. In order to accomplish this an Automated Guided 
Vehicle (AGV) system would be provided as shorn in figure 6. This systes would consist of 
battery operated carts that would be guided by a rail attached to the station structure. The 

carts would receive control signals via the rail and be directed to the specific location 
requiring the replacement part. These carts would be loaded and unloaded by a Automatic 
Retrieval and Storage (ARAS) systes as shorn in figure 7. This would assure that the pxo\.~ 
components were delivered in a timely Banner and would also support simultaneous servicing *: 
the subsystems. The AGV would also be useful in delivering equipment and tools to tt. 

teleoperated manipulators and astronauts during EVA. 


FIGURE 6 

AUTOMATIC GUIDED VEHICLE (AQV) 
FOR DELIVERY OF ORUS 






PI SURE 7 


AUTOMATIC RETRIEVAL AND 
STORAGE SYSTEM FOR ORUS 






Crossing ths alpha joint will bs a major problem in sarv icing cosponants on ths powsr 
ganaration booms. Ona altamativa is to stop tha rotation of tha alpha joint during tha time 
tha remote manipulator or ACT is crossing tha joint. This, howavar, raquiras additional powar 
to stop and start tha joint. To ovarcoaa this problam a Transfar Carriaga could ba providad 
as shown in figura 8. When tha ACT arrivas at tha alpha joint tha carriaga would ba locked 
onto tha Spaca Station structura . The ACT would mova onto tha Transfar Carriaga. Whan tha 
powar boon rotatas into position tha Transfar Carriaga would lock onto tha powar boom and 
disangaga from tha structura. Tha ACT would than mova onto tha powar boom. 

FIGURE 8 

AUTOMATIC GUOED VEHICLE (ACT) 

TRANSFER CARRAGE FOR ALPHA JOINT 



8. POTENTIAL APPLICATIONS 

Various subsystems on the Space Station are candidates for dedicated robotics. Two typical 
applications would be the Laboratory module and the Propulsion unit. 

PROPULSION 

A typical concept for servicing components on exterior system of the station is shown 
figure 9 for the propulsion system. In this case the robot could exchange ORU's consi sti ng of 
three propulsion units. The robot would also perform regular inspections of the propellant 
lines and fittings to check for leaks. A modular end effector could also be developed which 
would fit around a leak in a section of tubing and repair the tube in-situ. 

FIGURE 9 

PROPULSION SYSTEM SERVICING 
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LABORATORY MODULE 


As shown in figure 10, several dedicated robots can be provided in the laboratory module for 
servicing and operation of the experiments. This would reduce the cost to the customers by 
allowing them to automate their experiments without having to build it into their equipment. 
Instrumentation could be shared between experiments and customers thus lowerina the cost. In 
addition, customers could be allowed to control their experiments from the ground by using the 
OLP/s isolation facilities. Many experiments will be carried on in a vacuum environment. By 
servicing these experiments with a robot, the astronaut would not be required to suit-up. 


FIGURE 10 

DEDICATED ROBOT IN EXPERIMENTAL BAY 



* 

i 
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9. CONCLUSIONS 

Modern factories today rely on multiple dedicated robotic devices to increase the productivity 
of their workers and remove them from the repetitive and boring manufacturing tasks. The Space 
Station can also benefit from applying this technology to servicing and maintenance. In 
addition with the proper thought to component design the possibility exists that the dedicated 
robot systems could aid in assembly. This would have the added advantage of allowing the 
assembly of various subsystems to proceed simultaneously and reduce the time to commission the 
Space Station. This does not suggest that the Space Station can be totally automated at this 
stage. There will always be tasks that require the direct intervention of the astronaut. 
However many well defined and repetitive tasks exist that would benefit from the application 
of a robot requiring a minimum amount of adaptive control. By applying existing technology as 
well as limiting there use to well defined tasks dedicated robots could be made available for 
IOC. 
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1. Abstract 

v "''A demonstration of NASA's robotics capabilities should be a balanced 
agenda of servicing and assembly tasks combined vith selected key 
technological experiments. The servicing tasks include refueling and 
module replacement. Refueling Involves the mating of special fluid 
connectors while module replacement requires an array of robotic 
technologies such as special tools, the arm as a logistics tool, and the 
precision mating of ORUs to guides. The assembly task Involves the 
construction of a space station node and truss structure. It will 
highlight the proposed node mechanism. In the process, the servicer 
will demonstrate a coordinated, dual arm capability. 

The technological experiments will focus on a few important Issues: 
the precision manipulation of the arms by a teleoperator, the additional 
use of several mono camera views in conjunction vith the stereo system, 
the use of a general-purpose end effector versus a caddy of tools, and 
the dynamics involved with using a robot with a stabilizer. 

Proposed industry space manipulators range in lengths from two to 
fifty feet. If the robot is a free- flier, the length of the arms is a 
function of its docking location in relationship to its cask. As a 
result, no one set of manipulators can do every job. One solution to 
the problem is a "reconf igurable arm." This concept requires a modular 
approach of assorted arm lengths and different size drives including 
special modular sections. An Integrated package of a hand, wrist, and 
forearm with actuators is an example of a modular section. 


Interchangeable end effectors and tools Is another facet of this 
concept. Each configuration is customized to the application. The 
demonstration will test some aspects of the "reconf igurable arm*' during 
Che technological experiments. 

The robotic servicer will be mounted on a pallet as an experiment in 
the shuttle bay. It will be integrated with a minimum of two Cask 
panels complete with knobs, connectors, switches, and doors. The pallet 
will be self-sufficient and able to test all the aforementioned 
capabilities. A portion of the tasks will involve the Remote 
Manipulator System (RMS) in picking up the robotic servicer. The 
demonstration will answer several important questions concerning a robot 
doing extravehicular activity (EVA) and non-EVA types of work. 

2. Introduction 

Space is a natural environment for a test bed of advanced technologies such as 
robotics. The most frequently described missions include repair, housekeeping, or emergency 
work. Eventually, servicing and assembly missions will be common occurrences. However, the 
technology is far from mature with several critical questions yet to be answered such as 
"what are the major components of such a robot and will it be able to accomplish its 
mission?" 

A mature servicing robot cannot be developed without some key intermediate steps. One 
of these is the subject of this paper, a shuttle bay experiment that merits some attention. 
This demonstration will be based and operated in the shuttle bay. The robotic servicer is 
an extension of the Johnson Space Center (JSC) work being conducted on their 
anthropomorphic- 8 ized robotic servicer, the Telepresence Work Station (TVS) [1]. 

This paper describes the system, including the robot and pallet structure. The 
versatile system will be able to test various servicing and assembly scenarios, and will 
test key technological experiments. 
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3. Concept 

Mounted on e pellet (Figure 1) , the telerobotic demonstration experiment will be 
conducted from the shuttle bay In an actual space environment. The pallet will contain all 
elements of the experiment with the exception of the operator's console, mounted In the aft 
flight deck. The pallet Includes a dual-arm robot, task panels, equipment rack, support 
electronics, tether and take-up reel, and robot latch mechanism and launch restraint. 

The robot (Figure 2) will be configured with dual arms, each with seven degrees of 
freedom (DOF). Their basic design will be based on the Protoflight Manipulator Arm (PFMA) 
with an advanced wrist (3 -DOF, concurrent axes) and additional Improvements at the component 
and subsystem level. This design will lower the risk associated with successfully fielding 
an advanced manipulator In a useful time frame [2]. Each arm will be reconflgurable to 
allow changes in length, drive strength, and end effector. This capability will provide 
increased flexibility and opportunity to determine optimum configurations for particular 
tasks. Additionally, an evaluation of a general-purpose end effector, as opposed to an 
assortment of specialized tools, can be conducted. 



The robot vision system will include a stereo camera set housed on a 3 -DOF mount, above 
and between the arm shoulder joints. In addition, a single camera will be mounted on each 
forearm to ensure a closeup view of the workspace If desired. How these mono cameras can be 
used in conjunction with the stereo system for enhanced task performance will be the focus 
of several experiments. 

A tether and take-up reel will allow the robot to leave the pallet (Figure 3) and 
perform tasks in conjunction with the shuttle RMS without having to rely on a self-contained 
power source. This also allows the manipulator control electronics to be reaotely located. 



Integral to the pallet will be the task panels, representing a wide range of 
activities. At least two of these will be mounted askew to permit testing with the robot 
attacned both to the pallet, or the RMS. These panels can be tailored to almost any task 
the manipulator would oe required to perform. Their size would permit the simulation of 
portions of actual flight hardware, ensuring a realistic portrayal of a known servicing 
t* ... Including obstacles to be encountered and avoided. 
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A nearby equipment reck will contain all elements of the desired assembly tasks. Having 
both within Its reach envelope, the manipulator can use this rack of coeponenta In 
conjunction with the task panels. Increasing the number and complexity of tasks that can be 
performed from a fixed location. The different tasks are systematically assessed for 
complexity before the demonstration by an Index based on notion primitives [3]. 

4. Servicing 

Servicing Is a primary Issue for any telerobotlc system [4]. The failure of a fuse on 
the Solar Maximum Mission rendered the hardware useless until the unit was serviced. Some 
key servicing scenarios Include module changeout (macing and donating connectors, fastener 
removal, precision alignment); Inspection, checkout, and calibration; manually deploying an 
appendage: and cable take-up and untangling. 

Moduli c-jngeout Is characteristic of any servicing mission. The Hubble Space Telescope 
has 12 prlx.trv orbital replaceable units (ORUs) , such as batteries, electronic boxes, and 
fuse plugs [5j. When replacing an ORU, Initial steps Include the demating of several 
electrical connectors (signal and power). This requires a delicate and dexterous series of 
hand /wrist movements. The ORU can be removed once the fasteners are loosened. The 
subsequent reversal of this scenario replaces the module. Caution must be taken In 
precisely aligning the module during replacement. The module changeout can be demonstrated 
on the pallet system. Behind the door on the task panel would be a generic ORU. The robot 
would open the door and remove the ORU, then reattach the module to the equipment rack for 
st rage. Conversely, the same ORU will be returned to the task panel. The experiment would 
be repeated with the robot fixed to its base and attached to the RMS. 

Another servicing task involves the deployment of antenna booms with failed actuators. 

In this event, a Jackscrew must be turned for manua 1 deployment. Affixed to the task panel 
is a simulated boos the robot must deploy by driving the actuation mechanism. 

In many situations, parts and tools may be tethered on a line that must be reeled In. A 
tangled line can cause many complications. In this Instance, the tether must be handled 
meticulously so as not to worsen the situation. In such a case, the manipulator must 
demonstrate a high degree of dexterity. A tangled tether can be put in one of the sliding 
drawers of the task panel. The robot must untangle the line, wind it up, and return it to 
the drawer. 

The last servicing scenario is refueling. The critical step in refueling Is mating the 
male fluid connector to its female counterpart, necessitating handling a cumbersome hose. 

For this demonstration, the female connector would be in the task panel. The male connector 
is reeled out of the equipment rack and mated to the task panel. 

5. Assembly Tasks 

Candidate assembly tasks to be performed by the shuttle bay telerobotlc demonstrator 
include those necessary tor Space Station deployment and assembly of future orbital 
platforms. Teleoperated robots could execute boring and repetitive assemblies that would 
easily fatigue a Space Station EVA crew member. Primary among these is assembly of the 
truss structure forming the keel of the Space Station. Elements of one design candidate 
proposed by Lockheed Corporation would be carried on the equipment rack and 

assembled/disassembled by the manipulator [6]. Figure 4 illustrates this truss construction 
mechanism and how the two basic elements are assembled. Dual -coordinated arm motions are 
required as the mechanism is designed to be operated with the gloved hands of an astronaut 
using no tools. This task would be conducted with the robot secured to the pallet, negating 
the requirement for an additional stabilizing arm. 



Figure 4. Truss construction mechanism 
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This experiment could also be repeated with the robot suspended from the RMS as a scans 
of assessing the effectiveness of a single-arm stabilizer. In this position, one are could 
act as the stabilizer, while the other performs the assembly of a truss element to a node 
rigidly attached to the equipment rack. 

Additional assembly tasks Include construction and Installation of radiator panels. 
Installation and connection of utility trays, and Installation of payload support equipment. 

Sample components from various tasks would be Included In the equipment rack. 
Additionally, several configurations of the same task could be tested to determine the 
optimum design for a teleopera ted environment. 

6. Key Technological Experiments 

In addition to assembly and servicing, there will be several key Issues studied. One of 
these, the dynamics of a stabilizer, will be demonstrated by the assembly scenario on truss 
construction already mentioned. 

By doing these real tasks, the teleoperator will have the opportunity to evaluate stereo 
viewing versus several mono views. As previously mentioned, the robot has a stereo vision 
"head" and supplemental mono cameras mounted on each arm. Optimizing the field o£ view for 
each of these is critical to ease cask execution. Equally Important is lighting. Both wide 
and spot beams will be evaluated. Understanding the limitations of the vision system Is 
very Important, and this experiment will surely point them out. 

The robot will negotiate the standard complement of switches, knobs, and doors on the 
task panel. Also represented on the panel is a selection of fasteners and connectors, 
including the peg-in-the-hole experiment with the peg tethered. Figure 5 shows an example 
of a generic Cask panel. These tasks will test the manipulators 1 dexterity In space. We 
will also be able to investigate task sequencing as well as accomplishing a "time study." 
Each task Is repeated with the robot both securely mounted to the pallet as well as being 
held by the RMS. 



Figure 5. Example of a task panel 


A major topic is the issue of modularity. Length is a major consideration in 
manipulator design. If the arm was anthropomorphic, it would be approximately four feet 
long, in contrast to the shuttle RMS, which is fifty feet long. With this large disparity, 
it can be proven that no single manipulator can do every task. The answer cooes from the 
application. The length of the arm Is dictated by its location with respect to the task 
location. For example, if the servicer is attached to a docking port ten feet from the 
task, then the arm should be at least ten feet long to be able to reach it. 

The logical solution is to have more than one size* arm. What we are proposing is a 
"family" of manipulator components, that when assemb 1 t would lend itself to a variety of 
Jobs (Figure 6) . 


86 


Smafl Torque (15 ft-toj 
Wedkim Torque (50 ft-toj 


Drives Family 


Large Torque (90 fMbj 


XL) Extra-Large "toque (300 fHbJ 


4hm Ce«^k« 

nfTn Oijnipil rvivy 


Sme> Length (2 It) 


Medium Length (4 ft) 
Long Length (8 ft) 
Extra-Long Length (25 ft) 


Figure 6. Family of drives sod arm segments 

By assembling these "families" of components, it Is feasible to construct several 
different arms (Figure 7) to satisfy an assortment of missions. The result is some 
standardization of the manipulator with no major standardization of the tasks to be 
performed. As a consequence, the manipulators could be easily upgraded to Incorporate new 
emerging technologies. 

• 7-OOF. 8-fl Long Arm 



• 7-OOF. 6-ft Long Arm 


ILL 


• 6-OOF. 50- ft Long Arm 


Ml s LsTsTsj 


K 2f M 



Figure 7 . Several different manipulators 

This concept Is flexible, a compromise between distributed and totally Integrated 
actuation; a hybrid ve would like to call "reconflgurable" (Figure 8). 


' Manipulator ' 

Options 


Integrated 


Higher Precision 


! Option 


Reconfigurabte 


Hybrid Integrated/ 
Distributed 


Distributed 


Serviceable 
and Maintainable 


Figure 8. Hybrid manipulator option 


The concept optimizes the manipulator by using both integrated* and distributed modules. 
One example would be to use a Procoflight Manipulator Arm (PFMA) at Marshall Space Flight 
Center with selected upgrades. The existing arm, from the elbow back to the shoulder, has 
distributed actuators. The lower arm would be an integrated module consisting of Che 
forearm/wrlsc/end effector. Vlch the sctuators built Into the forearm, the wrist could be 
compact and dynamics Improved. Figure 9 Is an illustration of two possible configurations. 




Figure 9. Two typical ana configurations 

The success of the reconfigurable arm is dependent on an intelligent controller, and a 
mechanical attachment scheme that Is easily connected or disconnected. The controller must 
be able to handle any arm configuration; from long to short and from simple to multijolnted, 
and adapt to changing inertias, frictions, and other drive parameters. 

Our concept will include one reconfigurable arm. The initial demonstration will Include 
reconfiguring a 4-foot arm {two 2-foot segments) into an 8-foot arm (two 4-foot segments). 
The ability of the adaptive control system to compensate for this increase in length will be 
tested on the task panel. The attachment scheme should be light but rigid. It will be 



The modular theme also affects the end effector issue. The majority of servicing and 
assembly tasks require the use of some tool in conjunction with the human hand. The hand is 
the most flexible gripper or tool existing. The same is true for the robotic hand. At 
times, the hand will perform as a tool; for example, unscrewing a loose bolt without a 
wrench. However, it would not be efficient for our robot to have a tool holding another 
tool. Figure 11, from the TVS study, shows a mechanism that interfaces to a variety of 
tools such as a gripper or a ratchet. This mechanism has power and signal channels. There 
would be a locking interface to the different tools much like a bayonet mount, with a 
built-in power taKeoff. This power takeoff is a single drive that actuates any tool that is 
compatible with its interface. As a result, the weight for a motor in each tool is 
eliminated. 



Figure 11. Power takeoff mechanism 
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Presented Is a Modular approach to the utilization o£ a variety of tools. The numerous 
tools will vary from a parallel gripper to a camera as shown in Fleure 12. This flexibility 
allows this modular end effector to perform in more roles than a single tool, such as a 
screwdriver, could. 



At this time, it is not conceivable that an articulated hand can be packaged in the same 
manner as the previously mentioned tools. For this shuttle bay demonstration, there will be 
two lower arm assemblies. One assembly consists of the lower arm, compact wrist, and a 
dexterous hand. The other assembly is identical with the exception of the dextrous hand and 
the addition of a power takeoff mechanism. Both lower arm assemblies will be mounted on the 
equipment rack. The various tools to be exchanged will be attached to a holder on the same 
equipment rack. 

7. Conclusion 

Experiments, such as the shuttle bay telerobotics demonstrator, are necessary for the 
realization of a mature servicing robot. There are too many unknowns associated with- 
assembly and servicing tasks, and as many as possible must be resolved to successfully 
transition from a manual environment to teleoperation and automation. 


This experiment will provide an ideal environment where several key Issues can be 
explored. Including the following: 

Advanced control schemes designed to enhance telerobotic operation. Their higher level 
control modes can improve manipulator dexterity, and ability to adapt automatically to 
changing task environments. It is felt these control schemes may help ease the 
time-delay problems associated with in-orbit activities controlled from ground 
stations. Simulated random time delays would permit this evaluation and comparison to 
more conventional schemes such as bilateral force reflection. 

Vision and lighting systems, and how they affect the operators performance. How the 
mono camera information can be presented and used in conjunction with the stereo system 
to aid the operator while working in a cluttered or cramped environment. Visual acuity 
is extremely Important when faced with complicated tasks and relatively little or no 
experience. 

- Even with an adequate vision system, the ability to maneuver the manipulator arms around 
obstacles and perform dextrous tasks in cramped quarters must still be proven. The 
pallet demonstrator does not have to rely on the RMS to provide this type of challenge, 
as it is capable of performing many complex tasks with the manipulator fixed In its base. 

Individually, many of these issues have been demonstrated or are being developed In a 
simulated environment. It now must be shown that they all can play together and perform in 
the place that counts: space. 

The different elements of the experiment must be carefully chosen so adequate experience 
will be gained where most needed. Appropriate hooks and scars will be built in for future 
growth and to ensure a system with a long, useful life. 

The shuttle bay experiment Is not mission critical. It is well constrained on the 
pallet and presents minimum risk to other shuttle payloads and crew members. Its most 
significant impact will be on the knowledge and experience gained, by industry, in all 
disciplines of robotics. 
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1. Introduction 


The definition of telerobotics (TR) has not ret stabilized nor made the atandard English 
language dictionary* I tend to use telerobotlce is meaning remote control of robots by • 
huvm operator using supervisory end some direct control. Thus, this Is en Important eree 
for the RASA evolving soece stetlon. By robot, I mean e in l oul e t o r / m ob 1 1 1 ty device w ith 
visuel or other senses. I do not nne anl oule t or s , is In many Industrial eu tone 1 1 on se t - 
uos, robots even If they cen be flexibly programmed: rather calling these programable 

man l oul a to re. lur own laboratory at the University of California, Berkeley, has been 
involved In problems In dlsolav of Inforaatlon to the huaan ooerator. In orobleas of control 
of remote manipulators by the human operator, and In c o m m an 1 ca t i on delays and band-width 
limitations as Influencing both control and the display. A number of recent reviews have 
appeared with discussions of the history of telerobotics beginning with nuclear olants and 
underseas oil rigs. 


2 . Three S lmul taneous Research Directions 

I believe that we should engage In triplicate or three way olannlng. It Is iioortan'. to 
carry out our research to accomplish tasks (l), with man a lone. If possible, such as In 5V4 
(ex tra -vehicular activities), (ll) with autonomous robots (*?.), and (ill) with telerobotles. 
By comparing and contrasting the research necessary to carry out these three approaches, we 
may clarlfv our present problems. 


There are problems using man alone. The 
space environment Is hazardous. It Is very 
expensive to have a man in space; RASA must 
have quite adequate cost figures obtained 
from the demonstration proleots that have 
already been accomplished with the shuttle 
program. We may also need a higher quality 
of performance than man alone can provide In 
terms of strength, resistance to fatigue, 
vigilance, and in meeting special problems. 
For example. If the space suit Is not of 
constant volume under flexible changes of the 
limbs, then a great deal of strength Is used 
uo lust in meintaining oosture. 


FIGURE 1: tBl PLICATE Puwiltt 

PROBLEMS KITH F3AU AtfiNF 

HAZARDOUS ENVIRONMENT: 

(SPACE $1*11 A* TO NUCLEAR PLANTS. UNDERSEAS i 

EXPENSIVE U.E. EVA IN SPACE) 

NEED INCREASED QUALITY IN 

STRENGTH 

FATIGUE RESISTANCE 
vigilance 

PERFORMANCE 


Problems with autonomous robots lie In 
our not having mastered the technology to 
build them and have them oerform satisfacto- 
rily. They are not yet available! tndeei. 
designs are not yet fixed and it is not 
certain how feasible they will be , esoec tally 
in terms of robustness and reliability. 

Therefore, we can see that telerobotles 
; s i viable leading edge technology. 
However- ail three directions snoald be 
intensively pursued in research and 
development, especially for the nett stages 
of the evolving space station planning. 


Problems kith Au t 2nqmcu-i Robots 
NOT >ET AVAILABLE 
DESIGN nOT 

feasibility sot CEVAIV 

RELIABILITY NOT TESTED 

THEREFORE: IS IS A VIABLE LEADING EDGE ’ECHNCLOGY 

-l- >3££ DIRECT :;*S 3H0LLD 3E 5L?®CR’ED *C« E.CLVIHG .-ACE 

STATION PLANNING. RESEARCH. AND DEVELOPMENT . 


5. Space station Tasks 


Dne of the major roles that M 45 4 can o l a y is to hypothesize tasks for the evolving space 
station. In this way ressarch regarding the design of telerooots to accomplish these tasks 
can be guided. For a list of seven groups of tasks see Figure 2. 
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FIGURE 2 : 

NASA SHOULD HYPOTHESIZE TASKS FOR EVOLVING SPACE STATION 


As I will consllir liMr, It la 
Important to distinguish bstwssn thoss 
tasks animus to ths NAGA/svolvln^ Sosos 
titlon and thoss with "Industrial 
pIvips" that will KCOiol lih 
svslopmsnt of n a w tachnolo^lss In 
hopsfully a superior fashion and thus 
snabls conssrvatlon of 111 l tad NASA 
rssourcss. 


4 . Probl a ss In TsI srobo tics 

This nsxt ssotlon of my talk, 
rsvlsws of Droblsms In ts l a robo t l o a, 
•III bs abbrsvlatsd. Ths rsvlaw Is 
dlvldsd Into oroblsms In tslsrobotlos 
ooncsrnln* displays, vision and othsr 
ssnsss (Flours ^) and oroblsms In 
tslsrobotlos dsalln* with oontrol and 
eommunloa tlon (Flours l). 


housekeeping 

LIFE SuPPORf SYSTEMS 

Inventory control, access and storage 
Record keeping 
Garbage disposal 


ttaifcoifla 

From space garbage 
From meteorites 
From traffic flow 


MAINTENANCE 


Satellite 

vehicles 

Space station itself 


CflNSTRUCTlON 

additional space station structures 


In saoh ssotlon. t start with basic 
propsptlss of ths human oosratop and and 
up with olannsd oaoabllltlss of 
autonomous robots. In bstwssn, I try to 
oovsr what knowlsd^s sxlsts now In our 
flsld of tslsrobotlos. ( S a a also 
companion papsr by Stark st al In this 
vo luma . ) 


Manufacturing 

Crystal growth, biopmarmaceuticals 


MQJHL 1 TY 

Automatic piloting 
navigation 
Path planning 


SCIENTIFIC 

LANDSAT TYPE image PROCESSING for AGRICULTURE 

meteorology 

Astronomy 

Human factors research 
Scientific record keeping 


FIGURE 5 1 Display Problems IflR th£ Hjjman operate 
Human vision LLV, mlv. hlv 
Display Graphics < raster/vector > 

On-ihe-screen enhancements 
On-the-scene enhancements 
other senses displayed; 

Inputs to oth'r senses 


h.o. Control of TR 
h.O. Special Control: 

preview, delay, bilateral, homeomorpmic control 

'.OCOMOTION (HUMAN, ROBOTIC); 

NAV | GAT I ON - -PATHWAYS 

Potential field algorithms 


perspective and Stereo Displays 

msk performance criteria 

helmet M' - 

*x; space constancy 

human Operator (H.O.j Performance 

fatigue, effort, vigilance 


FIGURE 4: CflHTRQi an£ Communication Problems fqr th£ Human Operator 

Basic Properties of h.o., especially for Eva task performance 
nerve, muscle, ag/at model* 

Sampled-data (SD) and adaptive control 
prediction, preview, optimal control--Kalmam filter 

H.O. Control of vehicles, manual control 


Robotic vision* 

LLV - Chips 

MLV - BLOCKWORlD AND HIDDEN LINES 
HLV - I CM, A| 


Nota: LLV Is lowsr laval vision, MLV, 
mlddla laval vision. HLV hl^har Isvsl 
vision. Including ICM. Ima^a comorss- 
slon by modslln? and At, artificial 
in ts l l l ?snca . 


HLC (high level control); 

Supervisory control 

Mt/L 7 I PERSON COOPERATIVE CONTROL; RCCi: FUZZY SETS 


Autonomous Robotic t ar > control 

Sensory feedback, adaptive control, ai 

^ftiots: AG/AT Is an a^onlst/anta^onlst musola 

oalr, raclorocalty Innsrvatad for fast movamants 
and co-contractad for oostura and Imoad-nca 
oontrol . 


92 



5 


5 - Industrial Ortysrs For Csrtsln Sscsssiry Spies Station Tsehnoloj Iss. 

This n**t ssctlon dsils with ths futurs. md ssoselslly with "Industrlil drlrsrs" othsr 
thin WASA for nsw tsehnolo*lss whleh sir bs rsqulrsd In ths svelyln* Soms Stitlon. In Flours 
5 I list nlns cowoonsnts of i tslsrobotles systsw thit esrtilnly sssw to bs drlvsn by 
laportint Industrlil hirdwirs rsqulrsssn ts. rsssireh md dsvsloowsnt. Thsrsfors, It ss-ws 
rsisonibls for SASA to sit bis* md wilt for md syiluits thsss dsvs 1 ooisnts. sowing Its 
rssourcss for thoss nsesssiry tsehnolo^lss thit will not bs so drlvsn. 


FIGURE 5 : Drivers Qrm Thaw MSA £08 ft! 8 £ wffpfp T£CHWOLCgi£S 

ROBOTIC WANIPUt ATOR MC CONTROL SCHEfC 
JOYSTICK - AIRCRAFT 

Aft manufacturing Industry. Nuclear industry, mining 

Industry, Sensors: force and touch; compliant control 

52 v taa mobility 

military, tanks and other vehicle plans? 

UNDERSEA ROV - OlL AND COFWUNICATIONS INDUSTRY 

Locomotion - university Research 

Shipping Industry: Ships at Sea (AR. TR. Kan] 

IV Camera 

Entertainment industry - commercial device 
Security Industry 

meed MOUNTS, controls and MOTORS FOR PAM, TIL T 
AND FOR STEREO VS 

fig 

Entertainment industry is a better driver than 
COMPANIES BUILDING FlIGhT SIMULATORS; 

AS AN EXAMPLE. 

EM SENSORS RESEARCH/ HEAD-EyE MOUSE 

IQ! 

LANDSAT 

Security 

medical Industry - CT and MRI 
Industrial production lines 
HJ * Image understanding 

CreuCL*. 

Computer Industry 
(HDN) and ISP*) 

Computer Science =e search base is now very broad 

Cy. 'M CAT ^N 

Communication Industry is huge 
Shi»s at Sea 
B w compression 
Remote CJiu pigs 
arctic Stations 

Plans and ? »Q t ccol 5 Z 2 cqmba: h ( Q^ "at;g^£ anq 
12 r a LMQ TE ;-3I ANCE 

Op- f CE Automat :dn Forces 

A;p 'aapc.'C Ion's _ NEEDS 

Security industry 

Cqcpesawe Control 

MILITARY * SUBMARINE CONTROL 
H€LICOP t ER FLIGHT CONTROL 

Air TRAFFIC controllers 
nuclear industry 
Chemical Plant industry 


Looking it thsss flours* ^Itss us sews 
eonesot of how Industrlil dswsloossnt wit 
orowlds wirlous typss of tsehnolo^lss for 
ths sTolwln^ Spies Stitlon; Indssd. WlSl wit 
bs ibis to nick in d shoos* frow off-ths- 
shslf Itsws! For striot s. ths wost oowsrful 
cowoutsrs on ths list spies ihuttlss wsrs 
ths hmd-hsld oortibls, oowoutsrs thit ths 
istromuts brou^h* iboird which contilnsd 
such ^rsitsr cioi llty thin ths on-bcird 
cowoutsrs; thoss hid bssn ’rczsn In thsir 
dssl^n tsn ysirs i^c In ths dinning sti*si 
for ths sdics shuttls. 


6 . dscsisiry Tsl srobot 1 cs Tschno 1 oy 1 si tc 

bs S pi rksd by HU 

Hcwsvsr, thsrs i r s 3 »vsnl i r » i s in 
tslsrobotlcs thit wit llksly not bs drirsn 
Indsosndsntly of M \ S K , or whsrs NISI wit 
Hits in Iwoortint rols to dIit. Indssd. ths 
Con^rsss his sosclflcilly wmditsd thit !*}< 
of ths Soics Stitlon bud^st should bs ussd 
for Autowition md Robotics dsvs 1 oowsn t. md 
thit this In sows ssnss should sosirhsii 
Industrlil robotics In ths T Jnttsd Stitss 
(Flours SI. 


FIGURE 6: areas Sparked fiv nasa not Industrially Driven 

visual Enhancements for Graphic Display 

Telepresence with Stereo helmet Bunted Display ihwdj 

Kjlti sensory Input Ports: 

*orry about h.O. overload condition 

(ESPECIALLY WITH COCPERArr E CONTROL AND COWUN ICAT ION « 

higher level Robotic vision: 

Example-Image Compression by modeling dCK) 

(TO REQUIRE LESS INFORMATION FLOW AND FASTER UPDATE ) 

Special Control modes for h.O. 

homeomorphic control 
Bilateral control 

Time delay AND preview control for T!f^ delay 
Compliant control 

higher level Control languages 

<suo as RCCL: Ruzr* control; path banning by potential 

FIELD CONSTRUCTION* 

PywC’E CPROATISG ’/EhJCLES ' RCV > SPECIAL CONTROL PROBLEMS: 
NAVIGATION. ORIENTATION. OBSTACLE AVOIDANCE FOR RJV 

cooperative control: 

COOPERATION AMONGST hlManS. 'ELERCBOTS. AND AUTONOMOUS »CBCTS 

Compliant. Flexible, homeomcrpm;c *an:pvla— «s 
Grasp versus TOOL USING 

HowDMCRPHic Dual Wide Ccntrcl 

Impedance Control 
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7. Pntyerst ty Research 

t now would Ilka to mmke a plea that *AS\ should exoand and stimulate telerobotfcs 
research conducted within tha university environment. Of course. aa a orofsssor I 11 r have a 
blaa In this direction and I aa willing to llstao to contrary arguments! tn addition to tha 
banaflts of tha rasaaroh sooomollshed by un 1 ve rs l ties, RASA also ^ts tha education and 
training of naw engineering manoower specifically dlraotad towards talarobotlos. and focused 
on tha evolvln* Soaea Station. 

tfhat kind of university and educe ttonal rasaaroh should ba fundad In general by RASA. T 
believe thara ara two levels of coTt C with however thraa directions! Into which thaaa 
educational rasaaroh labs should ba classified. 


(1) First ara Sim ulation Te l arobo 1 t os La bor s torla a. Here wa naad *raohlcs computers, 
perhaps loystlcks. perhaps hi *her ^TeVeT. supervisory oontrol lsn*ua*es, oanarss. lms*e com- 
orasslon techniques and communlcs tlon sohaaas. I would *uess that our country naada at laast 
thirty suoh systaas for aduoatlon and training. Thasa systaas should ba wary Inexpensive, 
aoorox 1 ma tely $*50,000 aaoh. Thay naad not awan ba paid for by USi. slnca unlwarsltlas can 
orowlda suoh rasaaroh simulation laboratorlas out of thair aduoattonal budgets or from small 
Inilwldual rasaaroh grants. Our Talarobotlos T Jn 1 1 at Berkeley has baan thus fundad. A *ood 
daal of exploratory rasaaroh oan ba oarriad out Inexoenst waly In this mannar. 


(ill Saoond. wa naad Talarobotlo Laboratorlas with ohy s ioa 1 ma n 1 ou la tors orasant as 
tmoortant rasaaroh ooaoonants. In this waw. experiments wlt”varrous" robotic 'manloula tors, 
aaoaoially thosa with soaolal oontrol oharao tar l s tlo s suoh as flaxlblllty, hosaomorohio form, 
naw developments In *rasoers. and warlabla imoadanoa oontrol modas. othar than ara found in 
standard industrial man ioul a tors, would ba oossibla. I *uess that thara ara about flva suoh 
laboratorlas In soma sta*e of davaloomant at malor unlwarsltlas in tha country. I would 
further astlmata that thasa laboratorlas could aaoh usa an Initial davaloomant bud*et of 
$500,003 to anabla tham to ourchasa naoassary hardwara in addition to softwara as axlstant In 
tha Simulated Talarobotlos Laboratorlas. 


Another sat of costly laboratorlas would ba Talarobotlos Laboratorlas with ra mo ta 
ooa ra torin* wah 1 c 1 as (30V) . Here a*aln. wa naad about flva laboratorlas at unlwarsltlas with 
fTrsl class en*ineerin* schools. l^aln, I astlmata about $$00,000 aaoh for tha initial 
hardwara suooort of thasa =137 labs. Thar could than study transfar vahlolas. local Soace 
Station vahlolas, Hoon/Rarm Rovers, and avan compare RR T J vs. talarobotlo controllad wahiolas. 


Tha uoivarsltr laboratorlas would contrast with and sarwa a dlffarant function than 
on*oin* aerospace industrial laboratorlas. and RASA and othar government labora toriam. Thasa 
lattar assambla hardwara for damonstra tlon and faaslblllty studlas. Than unfor tuna ta ly thaw 
ara somahow unable to carry out caraful human factors rasaaroh dealln* with tha chan*in* 
desl*n of suoh olacas of a^uiomant. tn tha uolvarslty settln*. this aooaratus could ba takan 
aoart. chan*ed, revitalized. modiflad and tha flaxlblllty would inform our currant 
caoabllity. I would Ilka to contrast tha Sossamer Condor and Cossamar Albatross with tha 
$ \ 3 A oro*ram. tt was olaar that If doCraadv was avar to ba succassful, ha had to build an 
experimental olana which was axoactad to brack down aaoh axoarlmantal day. But tha olaoa 
could ba raoairad in a faw mtnutas! This "laboratory bench" ooncaot is so dlffarant from 
twen t 7 -yaar-ahaa d -ol annto* currently controlling our soaoa oro*ram that has baan affaotivaly 
allmlnatad at RASA. I think It Is tmoortant to ralntroduoa rou*h- and— ready flald 
laboratorlas back Into tha soaoa oro*ram. 


S. RASA Frizes 

Another rola that 1 A 5 A mi*ht olay Is to offar da mons tra tl on contracts or. avan battar. 
prizes for aooomollshmant of soaolflo tasks. A * a l n t turn to tha Kramer ? r l z e : hara a 

orivata individual donated prize monay to o e awarded to tha first to build a man-oowered 
aircraft conforming to cartaln car* fully laid out specifications. 

Communication channals for controlling remote vahlolas and ramota manio ulsters era 
alraady sat uo. Thus wa could havs ortze contastants demons tra tin* at dlfferin* locations on 
aarth at cna "*": naxt d a a on a tr a 1 1 o n s usin* alamants caoabla of ooeratln* to aoaca. or avan 

mora stringently, of having that minimum mass caoabla of bain* liftad Into soaea: and than *a 
ml*ht have true shuttla and soaea station dam ons tra ti ons. 


3 . 


Intallactual oroblams in Tl for tha Soaea Station 


Finally. I would Ilka to leava you with tha thought that tha list of to-be -soa rked -bv- 
RASA orcblam3 in Figure o contains many Important intallactual oroblams facia* tha araa of 
t a l a r obo t i c s. Although thasa araas ara bain* aooroachad in our rasaaroh community at tha 
orasant tisa, it may not ba oossibla to forsaa what noval kinds of challao*as will faca tha 
evolvin* Soaea Station In twenty years. Sven though I may not oradlct accurately. I certainly 
hooe t am there in oarson to watch talarobotics olarlo* a malor rola in ooeratln* tha Soaea 


Sta tlon. 


Ackoo w led*emant: Suooort from JFL 


Fills: and discussion with colleagues Crs. V. S. Kim. 


Or. Balczy; RASA - Ames (MCC-S51. 
Taodlck and S. Hannaford. 
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Robot Design for a Vacuum Environment 
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1. Abstract 
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Santa Barbara, CA 93116 
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' " The cleanliness requirements for many processing and manufacturing tasks are^becoming 
ever stricter, resulting in a greater interest in the vacuum environment We discuss the 
importance of this special environment, and the developement of robots which are physically 
and functionally suited to vacuum processing tasks. Work is in progress at the Center for 
Robotic Systems in Microelectronics (CRSM* **} to provide a robot for the manufacture of a 
revolutionary new gyroscope in high vacuum. The need for vacuum in this and other pro cesses 
is discussed as well as the requirements for a vacuum-compatible robot Finally, we^resent - 
details on work done at the CRSM to modify an existing clean-room compatible robot for use at 
- high vacuum. 


/- 


2. Introduction 

Among the many advantages of robots is their ability to work in harsh environments. 
Robots are being developed for maintenance of nuclear facilities (for example, the ODEX 
walking robots by Odetics, Inc) and for high temperature and other harsh environments. The 
high vacuum environment is now becoming more important in many high technology 
manufacturing tasks, and as a result the need for vacuum compatible robots is increasing. 

\ 

Most processes requiring high cleanliness standards are now performed in clean rooms. 
The principal users of clean rooms are advanced industries making use of thin film technology. 
Materials manufactured in ultraclean environments include [1]: VLSI semiconductors, compact 
discs, photographic films, magnetic and video tapes, precision mechanisms and sterile drags 
and antibiotics. Today's rigorous cleanliness requirements can be illustrated dramatically with 
the example of the actual development going on in the VLSI semiconductor field. Table 1 
shows that the critical particle diameter, i.e. the maximum size of tolerable contaminant particles, 
is projected to be 0.05 pm in the near future. 


Storage 

density (kbit/chip) 

Line spacing 
on wafer (pm) 

Minimum critical 
panicle diameter (pm) 

Matter dominance 
period 

16 

4.0 

0.4 

1981 - 1984 

64 

2.5 

0.3 

1984 - 1988 

256 

1.5 

0.17 

1984 • 1988 

1 x 10 3 

0.9 

0.09 

1988 - 1990+ 

4x 10 3 

0J 

0.05 

1988 - 1990+ 


Table 1. Line spacing and critical particle diameters for high density 
integrated circuits (adapted from [1]). 


As of 1985, the most demanding air cleanliness level established in the U.S. Federal 
Standard 209b is a cleanliness class 100. This refers to a maximum concentration of 100/ft 3 for 


* Research Assistant in the Department of Mechanical Engineering. UCSB 
t Faculty in the Department of Electrical and Computer Engineering. UCSB 

ttThe CRSM is one of the Engineering Research Centers established by the National Science Foundation 
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particles of diameter greater than or equal to 0.5 jim. It is proposed in [1] that the present 
standards be extrapolated as listed in table 2. 


Oeailinen Pvtides per ft 3 equal to and greater than: 


class 

0.02 pm 

0.1 |im 

05 pm 

5|un 

1 

10 3 

3x10 


• 

10 

10 3 

3x1b 2 

10 

• 

100 

t 

3x10* 

lO 2 

• 

1000 

t 

t 

10 3 

7xlOP 

10,000 

t 

t 

10 4 

7x 10 

100.000 

t 

t 

10 5 

7x 10 2 


* Indication not meaningful for statistical reasons, 
t Indication not relevant for the definition of clea n l in ess req uire m ents . 


Table 2. Proposal for extrapolating the US Federal cleanliness 
standards (from [1]). 

In the near future, cleanliness class 1 and even stricter environments will be required. The 
three main sources of particle contamination are: the outside air (10 7 - 10 8 particles > 0.5 pm 
m 3 ), equipment, and humans, who give off about 100,000 dust particles > 0.3 pm and more 
than 1000 bacteria and spores per minute. An increasing number of specially designed robots 
are being used in clean rooms not only because of their potential efficiency and productivity, but 
also to replace one of die principal sources of contamination: human beings. 

Eventually, a wide range of processing and analysis tasks will be performed in vacuum 
environments. The reasons for using this special environment are many and include the 
following [2]: 


• To prevent physical or chemical reactions occurring between 
atmospheric gases and a desired process; 

• To disturb an equilibrium condition that exists at room temperature so 
that absorbed gases or volatile liquids can be removed from the bulk of 
the material (e.g., degassing of oils and freeze drying), and adsorbed 
gases from the surface; 

- To increase the distance that gas and vapor particles must travel before 
colliding with one another so that a process particle can reach a solid 
surface without making a collision (e.g., vacuum coating and the 
production of high-energy particles); 

- To reduce both the number of molecular impacts per second and the 
contamination times of surfaces prepared in vacuo (e.g., clean surface 
studies and the preparation of thin films), and 

- To reduce the concentration of a component gas below a critical level 
(e.g., the removal of oxygen, water vapor and hydrocarbons in 
tungsten filament valves). 

Presently, epitaxial growth of semiconductor films takes place in the low vacuum range. 
Sputtering, plasma etching, plasma deposition, and low-pressure chemical vapor deposition are 
performed in the medium vacuum range. Pressures in the high vacuum range are required for 
most thin-film preparation, electron microscopy, mass spectroscopy, crystal growth, x-ray and 
electron beam lithography, molecular beam epitaxy, and the production of cathode ray and other 
vacuum tubes. [3] These environments are completely unsuitable for the presence of human 
beings. The special suits that are worn in cuter space, for example, would be much less 
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appropriate in a specialized vacuum chamber used for a critical processing task. The amount of 
tune needed to produce the desired pressure level once the suited worker had entered the 
chamber would prove to be quite long. The need for robotics and automated systems inside the 
vacuum chamber may thus be more urgent than in a normal environment or clean room. A 
principal advantage would be the ability to manipulate objects within the chamber without 
opening the vessel and subsequently re-evacuating, thus avoiding a very time consuming 
process. The availability of vacuum-compatible robots will improve the efficiency of many 
processes now carried out in vacuum and should encourage the use of vacuum processing in 
new areas. 

There are a number of differences between the vacuum environment of space and that 
produced artificially on earth. Pressures can be achieved in vacuum chambers which are 
comparable to those in standard space orbits. Pressure levels reach 10** and 10* 9 Torr at 200 
and 800 km above sea level, respectively. However, in vacuum chambers on earth, the 
outgassing from components in the chamber works directly against the pumping equipment In 
space there is no problem maintaining the pressure, but it is known that the outgassing of the 
space vehicle causes an expanding gas cloud to surround it The shape of this depends on the 
venting paths of the hardware, the outgassing of external surfaces and backscattering by the 
local atmosphere. Thus, outgassing is also a major concern in space. In addition, spacecraft are 
exposed to the full solar spectrum as opposed to only a fraction of the spectrum on earth. 
Possibly the most important difference between vacuum processing facilities on earth and those 
in space is the difference in the gravitational force. Some processing tasks may be better suited 
to the weightless environment of space. 


3. APPLICATION: Gyroscope Assembly in High Vacuum 

Delco Systems Operations in Santa Babara, CA has developed a revolutionary new 
gyroscope, which is now entering the production stage. For reasons described later, the 
assembly must take place in a high vacuum environment of 10*® Torr. The CRSM is modifying 
an existing robot so that it can operate at high vacuum. It will be used by Delco for production 
of the new gyroscope. 


Gyroscope Description 

The gyroscope to be assembled is the hemispherical resonator gyro (HRG) which Delco has 
been developing since 1975. The HRG is not a laser based gyro, yet does not have the rotating 
parts usually found in mechanical gyroscopes. Edward Loper and David Lynch of Delco list the 
following attractive characteristics of this gyroscope[4]: 

• Extremely low power dissipation, and hence * 
negligible warm-up transient 

• Passive mechanical integration of angular 
rate, but with whole angle readout, making it 
immune to electrical power interruption, and 
thus giving it high tolerance to nuclear 
radiation effects 

• Ability to operate at very high angular rates 
without perfor m ance degradation 

• Capability to operate over the military 
temperature range without temperature 
control. 


The HRG consists of three principal parts 
constructed of fused quartz as shown in Figure 1. 



Figure 1. Principal Components 
of the HRG (from [4]). 
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These parts are hooded together with indium alter being positioned relative to each other 
with accuracies in the sub-mkron range. The principle that the gyro operates under was first 
describe d by OJL Bryan, who did ex pe rim ents on the vibrational modes of shells of resolution 
using wine glasses in the hue 19* century. The basic operation of die HRG is represented in 
Figure 2. The "wine glass" in this case is the hemispherical part of the resonator, which is 
forced to vibrate at 2500 Hz, resulting in a standing wave with a zero- to- peak flexing amplitude 
of 4pm as indicated in Figure 2. As the entire gyro (all parts are fixed relative to each other) is 
rotated through an angle, the vibration pattern responds by processing relative to the resonator 
through an angle p roportional to the resonator's rotation angle. The location of the ribration 
pattern is sensed by electrodes located in the pickoff housing. By determining the movem en t of 
the vibration pattern relative to the case, and knowing the relationship b et w ee n pattern 
movement and case rotation, the angle of rotation of the case is found. More technical details 
and test results may be found in (4), (51 and [6]. 

Need for a Vacuum Environment 

The main components of the gyroscope are constructed of fused quartz, which is an inert, 
sable material having low thermal sensitivity and extremely low damping. This makes it an 
attractive material to use in precision mechanical and optical devices. It also exhibits extremely 
low internal damping, the damping time constant of the resonator being around 900 seconds. 
This means that very little energy is needed to sustain the vibration and that a power failure will 
not cause a loss of positional information until approximately 15 minutes have passed. The 
positive aspects of using fused quartz will deteriorate as the quartz becomes less pure. Even 
gases which attach themselves to the surface of the HRG components will cause variations from 
optimal gyroscope pe rf or ma nce. The gyroscope thus must be manufactured in a high-vacuum 
environment and maintained at such throughout its life, a minimum of 20 years. 

Gyroscope Production 

In order to achieve a relatively high production rate for the gyro, Delco has proposed the use 
of a large vacuum chamber as shown in Figure 3. There will be a large central chamber 
surrounded by 12 smaller compartments, each of which may be opened to either the outside or 
the central chamber. The central chamber will be constantly maintained at high vacuum. 
Ideally, then, the raw gyro parts would be placed into compartment #1 and be removed 
sometime later as a completed gyro, with the intervening assembly tasks performed 
automatically. One alternative to this is a smaller vacuum chamber in which all steps of the 
assembly would be performed. This would result in an extremely slow assembly task due to the 
continual and time-consuming actions of venting and re-evacuating the chamber, border to 



Figure 2. Inertial Sensing Mechanism of the HRG 
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Figure 3. Vacuum Chamber coocept for HRG Assembly 

take advantage of the increased manufacturing efficiency die larger assembly chamber would 
bring, a means of transferring gyro parts b etween various stages of assembly is required. A 
robot posit io ned at the center of the assembly chamber is the idol c o m p on en t 

The availability of vacuum-compatible robots is presently limited, although this is likely to 
change in the near future. A prototype vacuum-compatible robot has been developed by 
Yaska wa of Japan, but the small size of this robot eliminates it from consideration for use in the 
Delco system. As research progre ss es on the development of vacuum-compatible robots, Dclco 
has decided to have the CRSM modify an existing robot for use in their assembly task. The 
motivating factors in this decision were cost and time. Although it is desirable to obtain a robot 
which was designed and built specifically for the vacuum env ironme nt, the first step is to obtain 
a vacuum-compatible robot This will occur more rapidly by doing a modification to an existing 
robot When a ’ground-up' vacuum-compatible robot becomes available it can then be 
co mpar ed to the modified robot 

4. Description of the GMF E-310 Clean-Room Robot 

The robot undergoing modification at the CRSM is a GMF (General Motors • Fanuc) 
model E-310 cylindrical coordinate robot originally desi g ned for use in clean rooms to class 10- 
This robot was chosen for its size and configuration as well as its good accuracy and 
repeatability for a robot of its size. Tests at the CRSM have shown the repeatability to be 
ilOpxn. The E-310 used for the gyroscope assembly task has four degrees of freedom (see 
Figure 4) consisting of two linear axes and two rotational axes. The robot has three main 
housings which are joined by the shafts of two linear axes. The base housing contains the 
motors for the Z-axis and the Theta-axis. The Z-axis motor, through a belt, drives a lead screw 
• linear bearing - linear guide assembly which causes the Z-axis shaft to move vertically. An 
electromagnetic brake is installed on the top end of the lead screw to prevent the robot from 
dropping when power is not available. The Theta-axis motor is linked to the robot base through 
an RV gear reducer and a pair of spur gears. A cross roller bearing is used to support the robot 
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and allow f or ro tation. The R-axis boosing sits atop tbe Z-axis shaft and contains a mechanical 
confignarioovetysiinSar to that of the Z-axis. A motor mounted In die boosing is coopled to 
the Mw* 1 * R-axis shaft through a lead screw, driving it back and faith. At the end of this 
daft it the wrist housing, camming t fourth motor which drives the end-effector mounting 
ptateCAlphwmds) vis anghtangje gear set. 



IE3 

MOTION 

RANGE 


z 

0-300 mm 

300 wmh 

e 

-150* ~ +150* 

90*A 

R 

0-500 mm 

900 innA 

a 

-1W-+1W 

90*A 


Figure 4 . GMF E-310 Robot configuration and specifications. 


The task for the res earch ers at the CRSM was then to modify the GMF E-310 so that it not 
only could operate in a vacuum environment to 10 4 Torr, but could also operate over long time 
periods without degrading its surroundings (outgassing, etc). The design requirements called 
for a modified robot with capabilities as close as possible to the original robot Tbe motion 
range of the two rotational axes would definitely be needed for the assembly task in the vacuum 
chamber. As for tbe linear axes, it was felt that their travel range could be restricted if 
necessary. The dimensions of tbe final vacuum assembly chamber had not been finalized at the 
banning of the project and could be adjusted based on the final specifications of the modified 
robot It was e x pect e d that tbe final assembly chamber would not require the full 300mm of 
7r Axis stroke. The stroke of tbe R-Axis would be related to the horizontal depth of the vacuum 
sub-chambers surrounding the main chamber. The robot must have enough horizontal stroke to 
reach into these c h a mb e r s and p e rfo r m specific funcrioos. This is also dependent upon the final 
configuration of the end-effector, which might also extend into the chamber. It was agreed that 
an R-Axis stroke of at least 300mm would be desirable. An important point, however, is that 
this stroke must be useful when tbe robot is configured in the vacuum chamber. In other 
wads, when the robot rotates inside the chamber, the wrist should just clear the inner wall. 
Then when die R-axis is extended, die wrist will move into the compartment with full stroke 
capability. 

The completed robot most be capable of operating in te mp eratur es as high as 100°C. 
Although some of the compartments will reach temperatures much higher than this, the central 
chamber will not. An initial bakeout may raise die temperature of the robot to two or three 
times its maximum operating temperature. This bakeout need not be accomplished in a matter of 
hours since the robot will remain in a vacuum atmospher e as long as equipment service inirrvals 
will allow. A degassing cycle as long as one to three days is thus acceptable. 

hi summary, the principal design requirements for the modification of die GMF E-310 robot 
fa vacuum compatibility were: 

• Modification of axes movement range: 

• Z-axis: maintain 300mm stroke if possible 
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- R-axis: maintain 500mm stroke if possible; if reduced, 

resulting stroke most be useful in the vacuum chamber 

- Theta-axis: main t ain ±150° rotation 

- Alpha-axis: main t ain ±150° rotation 

• Limit negative effects on the vacuum e n v ironm ent (out gassing, etc) 

• Design far £ 100°C operating e nvir onment 

• Conyl ete the modifications within one year. 

5. Modific at io ns for a Vacuum-Compatible Robot 

The design of robots for vacuum brings together many disciplines, including the study of 
kinematics, dynamics, control, sensors, mechanics, materials, and tribology. Tribology, the 
study of friction, wear and the application of lubricants, is critical when applied to 
vacuum-compatible robots. Traditional lubricants have vapor pressures which are much too 
high for vacuum applications, resulting in rapid evaporation and unprotected surfaces. Dry 
lubricants, especially MoS 2 , have been used in many high vacuum applications, however they 
tend to generate more debris and need replenishing more often than wet lubricants. Advances in 
both dry and wet lubricants are being made rapidly, helping to make this important part of the 
robot design leu complex. 

The first decision in the modification of the GMF E-310 was between two basic 
p hil osophies. Theriot could either be totally exposed to the vacuum environment or it could 
be sealed in a type of "suit’’ which would allow the inside components to operate at atmospheric 
pressure, as they were originally designed to do. In order to expose the entire robot to a 
pressure of 10"* Torr, a number of key changes would have to be made. The major ones would 
be in the lubrication systems, the surface finish and materials, and the motors. After examining 
this choice, it was concluded that it would entail a substantial amount of redesign work, and that 
a total exposure robot would be better designed from scratch. The goal then became one of 
designing a new housing for the robot which would seal it from the vacuum environment, while 
accomplishing the design goals as set forth in section 4. The sealing "suit” has to be as kaktigfat 
as the walls of a high-quality vacuum chamber, yet must also allow the desired movement by 
sealing two linear (R and Z) and two rotary (Theta and Alpha) motions. 

Rotary Sealing 

A differentially pumped 360° rotatable platform from Thermionics was chosen as the 
rotational sealing mechanism. As shown in the cross section of figure 5(a), the platform 
contains three spring loaded seals which are 80% teflon and 20% graphite. Two chambers are 
formed which are pumped to different levels of vacuum. For this application, the chamber 
closest to the atmospheric pressure side is roughed to approximately 10" 3 Torr, while the 
chamber closest to the vacuum side is maintained below 10* Torr using an ion pump. Figure 6 
indicates the placement of the rotatable platforms in the design. 



(a) <b) 


Figure 5. (a) Differentially pumped rotatable platform for sealing Theta 
and Alpha rotations, (b) Stainless steel bellow for sealing R and Z linear 
motions. 
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The two linear axes are sealed with vacuum compatible stainless steel bellows (fignc 5b). 
Figure 6 shows how they are configured on the modified robot Note that an additional bellow 
has been added to die tear of the R-axis boosing. Because die robot internal p re ssur e is 
essentially 760 Ton 1 above its external envi r o nm ent, large forces exist which tend to posh die 
R-axis f or war d and the Z-axis upward. For the bellow sizes which were chosen, the upward 
force doe lo the pressure differential is 477 lbs and die o u tw ar d force on die R-axis would be 
232 lbs if no co mp ens a t i on were used. The rear bellow shown in figure 6 surrounds a pipe 
which is attached to the same linear bearing as the R-axis. Thus, a force is produced which 
dneedy cou nter ac ts the one lending to force the R-axis o utwar d. As for the Z-axis, the weight 
of the upper components tends to c o unt e ra ct die upward force sufficiently. 



Figure 6. Modified E-310 Robot 


Component modi fi cations 

The main housings, for the wrist and the R-axis, sre replaced with new housings which are 
designed to be vacuum-compatible and interface with the bellows and the rotary seals. A special 
feedthrough is designed for interfacing through die wrist to the end-effector. Otherwise, all 
internal compon en ts are left unchanged from the original E-310 robot. 


6. Future Work 

The vacuum compatible version of the GMF E-310 as modified by the CRSM is s c h edu l ed 
for completion in Spring, 1987. Before then, work will be done to develop appropriate 
vacuom-compatible end-effectors for use in high vacuum environments. A cooperative venture 
with Yaskawa of Japan will involve basic research into vacuum robotics and applications. 
Yaskawa has developed axial gqp pulse motors especially for use in vacuum. They make use of 
magnetic stainless steel for the motor body, have a special coating oo the coil, and have been 
specifically designed to p rev ent heat build up. Research is continuing into non-contact drive 
mechanisms which would allow the ehmiiiation of lubricants. 

7. Conclusions 

The modification of an existing robot for a specific task at high vacuum has been described. 
Vacuum processing will become increasingly imp or tan t as the need for ultraclean manufacturing 
grows. Vacuum-compatible robots will be necessary workers in this harsh environment The 
CRSM is developing robot and end-effector technologies to be used for vacuum processing. 
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1* Abttract 


Mult 1-1 imbed locomotion sy stew oporntlpg ' In a *hand over band** faahlon are 
attractive for the construe t Ion and maintenance of large structures In space. 
They offer much greater energy efficienpy as compared to thruster baaed mobility 
systems. They also free tbe designer of the structure from the necessity to 
Incorporate tracks or other elements necessary for construction robots into bis 
structural design. This type of locomotion svstem also offers great flexibility 
for handling unexpected situations such as structural failures. 


A veil developed technology of coordination of mult 1-1 imbed locomotory 
systems Is nov available. This pr esen tati o n tFtTtTBrtudn jesults from a NASA 

I sponsored study of several years ago. ^T bls Vj S a .. simulation study of s 

\ three-1 Imbed loc emotion /manipulation system. Each limb bad six degrees of 

1 freedom and could be used either as a locomotory grasping hand-holds . or as a 

l manipulator. Tbe focus of the study vas kinematic coordination algorithms. 


The presentation will also Include very recent results from the Adaptive 
Suspension Vehicle Project. The Adaptive Suspension Vehicle (ASV) is a legged 
locomotion system designed for terrestrial use which Is capable of operating in 
completely unstructured terrain In either a teleoperated or operator-on-board 
mode. Future development may Include autonomous operation. The ASV features a 
very advanced coordination and control system which could readily be adapted to 
operation in space. An Inertial package with a vertical gyro* and rate gyros 
/ and accelerometers on three orthogonal axes provides body position information 
I at high bandwidth. This is compared to the operator's commands* injected via a 

/ joystick to provide a commanded force system on the vehicle's body. This system 

Is* in turn, decomposed by a coordination algorithm Into force commands to those 
V which are In contact with the ground j The Individual leg controls are mode 

“switched “TJ»tV€fn a force control mode, when the foot Is on the ground, and a 
position velocity mode when the foot is being returned. This form of control Is 
attractive for space applications y6 f mult 1-1 imbed systems, whether for 
locomotion or manipulation, since tbi weight appears only as one of the forces 
acting on the vehicle body and/rhe coordination algorithms are set up to 
minimize generation of loads by ^fmbm pushing against one another. 



/ 


2. Introduction 


Mult 1-1 Imbed systems can be used for locomotion over space structures as well as for 
manipulation. A proven technology of artificial limbed locomotion Is now available l 1,2). 
A configuration which Is well adapted both to locomotion and to manipulation la shown in 
Figure 1 and has been studied In simulation [3,4 T. 


Limbed locomotion has several advantages for ua« over space structures. First, a 
limbed system can be wholly electrically actuated using only energy which Is renewable via 
solar panels. The actuation system can be configured for regeneration minimizing energy 
requirements. Further, limbed locomotion requires only discrete hand-holds and should 
require no modification of many structures. Finally, limbed systems provide great 
flexibility permitting adaptation to unexpected situations caused either by failures 
within the locomotion system itself, or by dsage to the structure It Is negotiating. 

Technologies which provide capability for autonomous Identification of hand-holds in 
real time f 51 are now under teat at several centers 11,6}. This raises the possibility 
that a remote operator need only designate a path which would be traversed autonomously. 
In much the same way as Is planned for a Mars rover {7}. In this manner, direct 
teleoperation would only he necessary when performing manipulative tasks. 
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In this presentation the status of the technologies needed for realization of a 
i Inbed loconot lon/nan lptilat Ion lystts will be reviewed. 

3. A Suggested Configuration 

As Is shown in Figure 1, the suggested configuration has three limbs. This will 
allow locomotion with two hands gripping hand-holds at all tines. It will also allow use 
of two manipulation nodes: a single armed mode In which two hands grip hand-holds leaving 
one free for manipulation, and a two armed cooperative mode with one hand gripping a 
hand-hold. It Is expected that manipulation would be performed In a teleoperative mode. 
Locomotion would be performed in either a teleoperative mode, or autonomously with the 
remote operator designating path segments, as mentioned above. However, even In the 
teleoperatlve mode, the operator would command only direction and rate. Individual limb 
movements would be fully automated. 

The optimum geometry of a limb for locomotion (8] Is quite compatible with that for 
manipulation T9]. In fact. If the first joint axis Is placed parallel to the longitudinal 
axis of the body (the preferred locomotion direction), the second axle intersects .he 
first and Is normal to It, the third la parallel to the second and Is placed exactly half 
way along the length of the limb, the fourth Is also parallel to the second and third, the 
fifth Intersects the fourth normally and the sixth Intersects the fifth normally, as shown 
in Figure 2. The configuration Is optimal for both purposes. However, a three limbed 
system lends itself better to a rotational ly symmetric configuration which favors 
omnl-dlreetlonal motion without a preferred direction. In such a configuration it is 
preferable to place the first axis of each limb parallel to the axis of symmetry, as in 
the Odex, for example [2]. 

A four-legged systes would have the attraction of allowing bilateral manipulation 
from a firmer base with two hands grasping handholds. It could also be configured as a 
bilaterally symmetric system with a preferred direction of motion, with some gains in 
locomotion speed and efficiency. TWL* would, however, probably only be an advantage on 
very large structures. The cost would be the necessity of coordinating 24 degrees of 
freedom rather than 18, and the corresponding Increase in system complexity. 

4. Coordination 


The primary problem In adaptive limbed locomotion is the automatic coordination of a 
relativelv large number of actuated degrees of freedom to achieve a desired body motion. 
A great deal Is now known about this problem as a result of the Adaptive Suspension 
Vehicle project fl). In particular, the problem of using Irregular, so-called "free 
gaits" to accomodate to sparse foot-holds or hand-holds is well understood. The 
architecture of a suitable free-gait algorithm is shown in Figure 3. With respect to 
gait, which is the phasing of limb movements, the situation in space is, in fact, much 
simpler than in terrestrial locomotion since there is no need to maintain stability 
against gravity. Thus, the sole determinant of the locomotion potential of a limb at a 
given instant is its kinematic margin [3). This problem was examined In the simulation 
studv described in references f 3 1 and (4]. 

The Adaptive Suspension Vehicle fl), shown in Figure 4, uses an inertial local 
guidance system to detect body motion at high bandwidth. The actual body rate is compared 
to the operator’s coimaands via a six axis joystick. The resulting error Is converted into 
a commanded acceleration and, hence, a commanded inertia force system. This inertia force 
system is combined with the weight of the machine and decomposed via a force allocation 
algorithm into commanded leg forces [8,9). Finally, these are decomposed via a Jacobian 
transformation into commanded actuator forces. The use of the legs as force generators in 
this fashion removes the necessity for sophisticated dynamic modelling of the legs. 

The inertial sensing package is a relatively simple unit consisting of a vertical 
gyroscope, rate gyroscopes on three orthogonal axes, and accelerometers on the same axes. 
Drift and integration errors are corrected at low bandwidth by use of position information 
from ; oirt position sensors on the leg joints. 

Obvious lv, a substitute would have to be found for the use of gravity for a vertical 
reference :n t . . i 3 scheme. However, the force control strategy is very applicable. .In 
this manner the problem of limbs working against each other, which is characteristic of 
cooperative, multi-limbed operations, is avoided. 

5. Sensing 

Although it can be assumed that the geometry of the structure over which the machine will 
operate will be known, except for circumstances In which the structure is damaged, it is 
still necessarv to provide on-board sensing to identify hand-holds and other features. 
This is necessary to correct location of the hand-hold for inertial system drift and other 
system errors. In fact, location of landmarks, perhaps combined with proprioceptive 
position sensor information, can be used to provide the necessary low bandwidth position 
data to update the irertial guidance system. As was mentioned above, technologies vhii»- 
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trill permit automatic, real-time Identification of each features are being developed In 
the Adaptive Suspension Vehicle (AST) (5J and Autonomous Land Vehicle (ALV) [6] programs. 
The scanning rangefinder described In reference (5] la being used to select footholds for 
the Adaptive Suspension Vehicle. It provides a range Image In angle-angle coordinates 
scanning a 128 X 128 pixel field et two frames per second. In the ASV configuration the 
field scanned Is 60* In the vertical plane end 80* In the horizontal. The ALV uses a 
scanning rangefinder vlth a slightly different field. Similar Instruments are being used 
at several other centers. 

Tbs necessary technology for conversion of the range image Into a continuously 
updated elevation map In Cartesian coordinates based on the vehicle, end for selection of 
footholds based on that map, has been developed for the ASV, end would be directly 
applicable to a limbed locomotion system in space. The scanner, in the ASV configuration, 
would be too heavy and bulky, and too expensive In power requirements for use In space 
construction. However, since this was the first operational unit, considerable 
development is possible. The mechanical scanning system could readily be redesigned Into 
s much more compact, and much lower power system. The need for higher resolution at 
shorter range would allow use of a lower power laser. 

Video based terrain modelling technologies are also being developed [6], particularly 
in the ALV program. At present, the emphasis Is cm Identifying large features, such as 
roads. Further development of this approach might ultimately provide models at higher 
resolution than can be achieved with the laser scanner approach. 

Another Important feature of the sensing system Is the need to sense both position 
and force at all actuated joints. In fact, joint rate sensing Is easily added and Is 
useful In limb control. Position, velocity, and force sensing are used on the ASV 
actuators (11, Posltlon/rate sensing Is Important for guidance of the limb when It Is not 
gripping a hand- hold, and for Inference of the position of the system from known 
hand-hold positions. Force sensing Is needed for limbs gripping hand-holds, and for 
cooperative manipulation. 

6. Controls 


A number of features of the control system for the proposed machine have already been 
described. The proposed system has a hierarchical architecture shown In Figure 5. It is 
based on techniques proven in the ASV project. As Is the case with that system. It is 
Intended that the operator could Interact with the system at several different levels 
(1,3). At the highest level, the operator would designate path segments to be 
autonomously traversed, as suggested In Section 1. Figure 6 shows a composite photo of a 
three-legged robot walking over a series of path segments (A], It might be noted that 
operation In this mode would require very little hardware or software added to that needed 
for continuous interaction. This Is a result of the need for automated hand-hold 
selection and coordination for effective locomotion. 

The next level of Interaction would be one in which the operator continuously 
comaumds body velocity and angular velocity during locomotion via a joystick controller 
similar to that used on the ASV [11. This Is expected to be an effective control mode for 
relatively short traverses. Note that hand-hold selection and limb coordination would be 
fully automated In this mode. 

Ac the next lover level of Interaction, the operator would directly control the 
position, relative to the machine body, of one, or tvo hands. NASA has investigated 
slx-axls controllers suited to this function [12] . This mode would be used both In 
teleoperated manipulation and In locomotion when the operator finds It necessary to 
directly control hand-hold selection and limb and body motion. This might happen, for 
example, when repairing damage. 

It should be noted that the proposed system has a great deal of inherent flexibility 
of operation and redundancy. Locomotion Is possible using only tvo limbs. Loss of 
function of several actuators could be accommodated to with degradation In performance, 
but without total loss of function. The use of differing mixes of automated and 
teleoperated function allows adaptation to a large variety of situations. 

7. Conclusion 


A technology of coordinating mult 1-1 Imbed systems for locomotion, cooperative 
manipulation, and for manipulation with multi-finger, multi-degree of freedom hands Is now 
available. It Is here suggested that a manipulation system vlth at least three dual 
purpose limbs can provide an attractive mobility capability. Key features of the 
technology for this purpose have been discussed In this paper. 
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ripurc 1. Proposed conf iguration of a limbed locomotion /manipulation system. 
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Optimal geometry for a dual purpose limb. 
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Commercial Users Panel 



INTRODUCTION 

The National Aeronautics and Space Administration (NASA) Office of Co mm e r cial 
Programs (OCP) has responsibility for insuring the transfer of NASA-developed and - 
sponsored technology to the public and private sectors. The OCP program of actively 
pursuing new projects for terrestrial application of NASA technology is assisted by I) a 
Technology Applications Team (TATeam) at the Research Triangle Institute (RTI) and 2) a 
network of Technology Utilization officers at NASA Field Centers. 

NASA technology transfer has been fostered by an original mandate in the Space Act 
of 19SS which created NASA. This provides for ’...the widest practical and appropriate 
dissemination of information concerning NASA activities and the results thereof.” More 
recent directives include 1) the Steve nsoo-Wydler Technology Innovation Act of 1980 (PL 
96-480), 2) Report 98-867 of the Committee of Conference to Accompany Bill HR 5713, 
which authorized funding for the Space Station, directing NASA automation and robotics 
to be ’.-identified and developed not only to increase the efficiency of the Station itself 
but also to enhance the nation’s technical and scientific base leading to more productive 
industries here on earth.” and 3) the Technology Transfer Act of 1986 which further 
promotes industry interaction with Federal laboratories. 

The Jet Propulsion Laboratory’s Space Telerobotics Workshop afforded an 
opportunity for the non-aerospace community to review NASA-related projects and 
planning in automation and robotics (AAR), an area of emphasis by OCP. OCP selected a 
Commercial Users Panel to meet at the workshop. This group. Table 1, represents 
organizations and industry sectors with the potential for expanding commercial 
telerobotics. 

In contrast to the aerospace and academic makeup of most of the Workshop, the 
panel primarily represents the commercial sector outside the NASA family. While such 
industries as automotive or electronics have lesser motivation for man-in-loop approaches, 
they do represent about 70% of the market for robots in the U. S. It was also kept in 
mind that single component spinoffs (e.g.. sensors, system architecture, manipulators) from 
NASA telerobotics-related research could provide commercial improvements in any sector. 
Such representatives as the Ui. Army’s Human Factors Laboratory do not directly 
represent commercial interests, but their decisions and efforts will have important future 
impacts on automated systems, especially in mobility and manipulation. Such applications 
of NASA telerobotics as satellite servicing were not represented on the panel since these 
are so closely linked to NASA programs as to assure natural technology transfers. 
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Discussion by the panel was focused on telerobotks, fast because NASA has the 
opport u nity to remain at the l ead in g edge of this science through Space Shuttle and Space 
Station activities and through otter NASA RAD programs. Second, telerobotics represents 
a limited subset of the overall NASA aatoamtion and robotics activity which could be 
reasonably discussed within a limited timeframe. NASA has decided to utilize telerobotks 
as a foundation for its AAR activities and to move toward more autonomous systems from 
this foundation. This approach is also strongly influencing Strategic Defense Initiative 
(SDI) planning, especially as related to unmanned maintenance of orbiting platforms and 
satellites. 

Commercial User Panel members examined NASA-related telerobotks programs and 
technology through materials supplied before the Workshop, and through plenary sessions, 
conversations and otter presentations at the Workshop. The panel then met in closed 
session to discuss how, why, and where NASA developments might be applied. 

MOTIVES 

The NASA Technology Utilization program is predicated upon a demand side 
approach, Le., problems and requirements are requested from companies, reviewed through 
the Technology Utilization network, and matched where possible with appropriate NASA 
technology. The approach relieves industry of having to sort out the myriad technology 
developments over NASA’s history. Similarly, the CU Fuel was asked to identify needs 
of industry possibly related to NASA telerobotks developments. On the earth, several 
motives for teleoperation arise. These can reasonably be grouped into the categories of 
Safety, Security, and Productivity. 

Safety. The primary concept in promoting safety by telerobotks is that of allowing a 
remotely located person to manipulate, inspect, or perform some other action in a hostile 
environment The nuclear industry has long implemented master-slave type operations to 
allow Handling of radioactive materials. Related to this, the Savannah River Laboratory is 
currently supporting GCA to build a large telerobot to handle contaminated equipment and 
put it into storage. Both Dr. Naser and Mr. Byrd discussed how telerobotks are becoming 
an option to improve the safety of 1) nuclear power plant operations and maintenance and 
2) response to emergency situations. The essential precept is to minimize possible human 
exposure to radiation. In emergency situations, a telerobot can move into a Hostile 
environment much more quickly than a person who must wait for radiation levels to 
subside or at least be specially clothed against radiation and high temperatures (ice water 
circulation suits). The telerobot may be unable to completely solve a contingency problem, 
but may perform many simpler preparatory tasks. This allows a person to follow up and 
complete the more complex parts of a repair. An example is telerobot removal of bolts 
from a valve flange. The valve is then aligned and replaced by a human who leaves 
immediately, allowing the remotely controlled robot to complete retightening and initiate 
leak testing. 


Handling of other hazardous materials such as toxic c hem icals with totoaptiain 
manipulators has received less attention. Laboratory robotics are becoming more iotoortr/n 
in the chemical processing industry. Safety motivations for utilising toternhotka in f/dd 
operations (for example, die handling of hazardous wastes, from sampling through // m 
removal) have not resulted in a market large enough to attract aqmptoent asanufact/rers. 
Mr. Leach indicated that Caterpillar had discussed such systems with cnor mies 
responsible for hazardous waste management, but could not rede si gn equipment m less 
than a dozen units. 

Mr. Sebok of Perry Offshore and Dr. Schnakenberg of USSM said that their 
organizations’ respective ultimate objectives were to remove people from the water and 
from underground. Oceanographic explorations and off-shore oil rig aminteneace have 
more recently emphasized the utilization of unmanned sabmersibtes, a trend that is aimed 
finally toward near-autonomous systems. This trend and the advances in aadenea 
machine* have been in evidence with the recent explorations of the R.ML Titanic by the 
Woods Hole Oceanographic Insitute’s submersible, Alvin, and its tethered telerobotic 
"eyebair, Jason Jr. (National Geographic, Vol. 170, No. 6, December I9S6). 

In mining, not all hazards are associated with underground operations. Dr. 
Schnakenburg stated. For example, there is interest in automating high wall surface 
mining where the miner drives into a wall with a coal seam exposed and goes as far as 
possible, e.g., out to 1000 feet. Inexpensive but effective teleoperation is needed to 
optimize this process. 

Robotics have as yet made little inroads into the health and medical field, but 
considerations for staff safety, said Dr. Leifer, may move teleope rated units into eolation 
wards. Also, patients who must avoid exposure to any contamination (from other people 
or through frequent access and egress to their isolation rooms) could be remotely treated 
and supported. 

Dr. Hodge expressed the Army’s concern for promoting safety through AAR as one 
of 'soldier survivability*. Here telerobots might be used in activities ranging from 
removing fuses from unexploded armaments to initial engagement of an enemy. 

Safety was also considered in an aspect other than human protection. In the case 
discussed. Dr. Hollis indicated that the one possible application of telerobotics as a system 
in electronics/semiconductor manufacture would be in clean rooms. Contamination-free 
environments are preferred and sometimes essential for semiconductor work. Telerobot 
use would also reduce the size of these expensive enclosures, allow more flexibility than 
wire guided mobile units, and allow prompt emergency responce. 

Security: This category relates to safety but with an important distinction. i.e., rather than 
a requirement to remove people from hostile environments (safety), there is sometimes a 
need to maintain a person in the control loop to provide confidence (security) that 
adequate intelligence and experience is available. This is again perhaps most obvious in 
the nuclear utility industry. As stated by panelists Naser and Byrd, people are simply not 
going to be replaced by autonomous machines in nuclear power plants. 
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NASA considerations in applying telerobotics follow much the same lines. A 
teleo pc rated manipulator in orbit is probably handling very expensive equipment in a 
critical environment under conditions which can be only partially simulat e d on the ground. 
Here a "mistake* by n more autonomous machine could mean a failed mission or even a 
threat to personnel in orbit At the current stage of robotics development the level of 
confidence is greater for die telemanipulator with human backup. 

Another area where a high level of confidence is essential is in the health and 
medical Held. Mr. Flatan described a p ropos e d endocorpnscular teleoperator for endo 
surgery and other emerging options of teleoperator microsu r gery were discussed. Stanford 
and the University of Southern California are pursuing such research, the latter utilizing 
X-ray tomograms for manipulator orientation in brain surgery. Dr. Hollis also briefly 
described a project that IBM is pursuing with the University of California— Davis in 
robotic machining of bone for prosthetic knee or hip joint implantation. More precise 
routing should promote improved bone ingrowth. In all cases, the surgical physician must 
constantly observe these operations and maintain ultimate control. A consideration here 
related to telerobotics is how to provide a capability for failure or fault prediction and to 
allow near-instantaneous interruption by the human controller. 

The panel noted that NASA progress in telerobotics is important to the "security” 
category of motivation because NASA demonstration of technology can offer confidence to 
those in the commercial sector who must first minimize risk in applying automated 
systems. NASA applications should also help identify those parameters of teleoperation 
entailing higher risk than others. 

Productivity: Safety and security considerations have been the most important motivations 
for the industries (i.e., nuclear and undersea) which have led the way in telerobotics. 
However, these motivations may be closely related to productivity, and it was interesting 
to find that the panel emphasized increased productivity in considering telerobotics for 
commercial applications. This agrees with the needs for automation in space described in 
the original report by the NASA Advanced Technology Advisory Committee, "Advancing 
Automation and Robotics Technology for the Space Station and for the U.S. Economy" 
(NASA Technical Memorandum S7S66). Here productivity through AAR has been broken 
out to include — lowering of operating costs, increasing flexibility (communications, 
computers, modularity) to support innovation, improving reliability, achieving station 
autonomy, and performing tasks unsuited to humans alone. "Reducing hazards" constitutes 
a final category. 

As a first example by the panel, John Hodge said that, beside soldier survivability, 
their lab concentrates on 1) force multiplication, and 2) reducing military operation costs. 
Related to the (list area, the Army operates on no-projected-growth of manpower. High 
tech is being relied upon to increase productivity or control more weapon systems. Of 24 
current related Army projects, six deal with weapon delivery (vehicles, mobility), four 
with reconnaissance (mobility, sensors), and fourteen with services and support 
(manipulation and mobility about half and half)- He stated that his shop was supporting 
development of I) a field material handling robot (FMR), an autonomous robot with a 
twenty-five foot reach and 4,000 pound payload for handling ammunition, 2) a soldier 
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robot interface project (SKIP), which is a small teleoperated platform with a manipulator 
bavins a six foot reach and 150 pound payload, used for ordinance disposal and refueling, 
and 3) a single manned station for controlling two semi-autonomous vehicles 
simultaneously. This project is ca l le d TEAM. The Army is forecasting AI and robotics 
developments as they relate to their own projects and training. There is also work 
underway on a robot sentry vehicle which could patrol nuclear weapon stockpiles. 

Teaching robots efficiently requires addressing many of the same human interface 
problems found in telerobotics. Mr. Schmuter related one aspect of Ford’s automated 
manufacturing to telerobotics developments. To improve efficiency, his group is budding 
a generic robot controller for computer-aided path generation. The ides is to reduce by 
an order of magnitude the number of points to teach for 3-D complex trajectories. Most 
pendant buttons and the typical recourse to a special robot language would be eliminated. 
After thirty minutes of instruction. Ford employees are now teaching 3-D paths in about 
15 percent of the time it may have taken before. 

Caterpillar is also looking for better man-machine interfacing, said Gene Leach. 
They will accrue the same advantages as space based teleoperators from improvements in 
levers, knobs, linkages and such control strategies as going from joint-specific to end- 
point control. Heavy equipment vendors, both for surface excavation and underground 
mining, are increasingly seeking 'expert system*- type diagnostics for maintenance and 
fault prediction of machines. The panel experts for both mining and excavation believe 
their industries will keep the man on the machine for some time to come. The first 
priority of new systems is to make the job easier for the man, automating more repetitive 
cycles, within the context of cost effectiveness. John Hodge mentioned that the 
enthusiastic response to Army solicitations for field materials handlers indicated a market 
potential. 

George Schnakenberg explained that the underground mining environment is one of 
consistently structured geometry and that automation is most effective, in the long wall* 
mining prevalent in Europe while the "room-and- pillar” approach common in US 
underground coal mines introduces more complexities. Also, metals mining is so batch- 
process oriented as to be non-conducive to automation. Nevertheless, with 82 percent of 
US recoverable energy reserves being represented by our coal resources, more productive 
mining should be a national goal, and productivity is an important consideration in USBM 
automation planning. 

Thin seam mining utilizing TV cameras and teleoperators is under study, as well as 
automation of simpler repetitive tasks on continuous mining machines. Canada is 
attempting to develop automated drills and load haul dumps (2-8 yard scoops that roll on 
rubber tires) and Germany is attempting to automate installation and removal of roof 
supports. 

Nuclear and other utility power plants could operate more efficiently with properly 
designed surveillance, testing and preventative maintenance teleoperators. Such remote 
controlled duties as checking valves and sensors, looking for leaks and testing for radiation 
content of any leaks should also be accompanied by some on-board intelligence. As with 
any teleoperation involving routine tasks, operator strain from constant observation is too 


high, eg., long hours using a stereo monitor. Surveillance and scrabbling telerobots have 
received well-publicized testing at the Three-Mile Island nuclear plant and remote 
operated cavity cleaners are being used. The three major nuclear power utility vendors 
have used robots in the steam generator area. Both Germany and France have a brigade 
of emergency telerobotic vehicles on exhibit. However, Carl Flatau believes this 
technology lags US potential technological capability. He cited a unit developed mutually 
with a Belgian company which can climb stairs, clear obstacles 16 inches high, has 
autonomous capability (backtrack to operator) if radio contact is lost, and has 
programmable navigation including some obstacle avoidance. 

For Undersea operation, telerobotic system ruggedness and reliability is especially 
essential, first because of the difficult logistics of maintenance and secondly, because 
standby costs at off-shore oil rigs range from $10,000-100,000 per day. Mr. Sebok 
described their teleoperations (which can be applied down to depths of 20,000 feet or 
more) by two methods, 1) for simple tasks, multiple degree of freedom manipulators 
operating off subsea vehicles and 2) for complex tasks, specially designed tool and 
manipulator work packages as complicated as the vehicles themselves. For example, a 
package for a large oil company installs 650 pound insert valves with 30,000 pound torque 
castellated nuts. The package also removes valves, tests seals integrity, and replaces a 
3,000 pound, seven cubic foot control pod. Inspection and non-destructive testing are 
often applied. 

Dr. Leifer’s work with the Veterans Administration Rehabilitation Center has been 
concerned with applying telerobotics to increase the productivity and self-sufficiency of 
disabled persons. He indicated that for younger handicapped persons, investment in a 
$100,000 tool to get back into the work force is obviously attractive. Abo, he emphasized 
that physical therapy and other patient care is growing rapidly in the face of de m a n ds to 
push health care costs down. Acute (versus chronic) care is only 30 percent of all medical 
care. The therapy and chronic care market can be addressed by telemanipulation on the 
low-cost, low-intelligence end. A robot using force feedback has been developed by 
Athtec Corp. for human performance evaluation and will be marketed for health care. 
Otherwise, the panel was not aware of any company marketing a telerobotic product to the 
$S5 billion plus medical industry. 

As enlightening as the described potential advantages for telerobots were, IBM 
offered just as informative reasons for not using them. Dr. Hollis pointed out that the 
electronics/computer/semiconductor industry requires high speeds (approaching 100 (0 
motions per second), low tolerance precision and accuracy (microns or sub-microns), 
typical assembly intergrating 35-45 parts with different tools, and operation in a fairly 
large workplace. An automated, sensory-feedback error recovery capacity is necessary to 
deal with assembly process exceptions without informing an operator any more than every 
thirty minutes. One operator should be able to tend several robots. Within the industry, 
robots assemble while humans take things apart for maintenance and repair. While the 
automation trend has been somewhat slower than expected, IBM exemplifies the industry 
picture with its automation commitment This includes design and execution in 2- and 3- 
D assembly, geometric modeling, queing and scheduling systems, 2- and 3-D machine 
vision support, new actuation developments, and evolving computer architecture for real- 
time control. 
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Of Mcosity, • number of possible application areas for telerobotics were not 
repres e nte d by the panel. For example, not included were firefighting and agriculture. A 
survey before the workshop turned up intelligent-machines-in-sgricultuie efforts going on 
at the Universities of Georgia, Florida, Purdue, Calif ornia-Davis, and at Weyerhauser and 
Bate De-Nor th we s t. Studies covered soil analysis, planting seeds, harvesting, nursery 
maintenance and pruning. 

SPECIFIC NEEDS 

Beyond the general interests and more specific activities described under Motives, 
die panel indicated some items that might head a NASA telerobotics technology transfer 
shopping list. Not surprisingly, new und improved software received considerable 
attention — but not always specifically related to telerobotics. Some examples of these 
requirements include: 

1. Expert systems to d«agwn«e equipment condition for monitoring and predicting 
failures, combined with machine maintenance daring off-shifts. 

2. A CAD/CAM type simulation system for mining applications. 

3. Methods to reduce the coding required for handling exceptions in electronic 
assembly. 

4. Intelligence built into teleopera ted machines so that they know when they need 
help and can decide on proper response. 

3. More expert systems work on the design and planning end rather than on the 
analytic or diagnostics end. For example, a system searching through a large 
set of design alternatives, for associating them and arriving at a device 
synthesis. 

6. Scheduler/planners, an area where NASA excels. 

7. Data capture codes, to prevent production process decision information from 
being lost in host computers rather than being readily accessible to a localized 
plant requirement 

8. Emulators, which incorporate human factors simulation, as with cockpit or 
flight control simulation. The implications for telerobotics training and 
simulation are obvious. Recent examples are a Honeywell system for Army 
tank operators and a Mercedes automotive trainer. 

Joe Naser and Ray Gilbert discussed a software technology transfer project on which 
EPR1 and NASA are collaborating. The transfer derives from an expert system developed 
for the Space Shuttle at the NASA Kennedy Space Center. The original system was for 
liquid oxygen handling, which then evolved to a knowledge- based automatic test 
equipment system (KATE) for system monitoring, signal validation, fault location and 
d i ag n os i s, automatic control and reconfiguration. EPRI is taking advantage of this 'off- 
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the-shelT product to further develop h and apply it in s im u l a t ors and eventually in a 
nuclear power plant production system. Joe Byrd added that such software is needed by 
their industry but must be begun in simple applications and gradually assi m i l ated into the 
system. He also commented that it is often easier to incorporate new hardware than 
software for nuclear power. 

On the low end of teleoperator hardware. Dr. Leifer reiterated that the medical field 
will first best incorporate •dumb" systems which serve to extend a patient's control a little 
beyond his reach. However, voice control is a more sophisticated requirement often added 
to such systems. 

Mining, nuclear and undersea experts stated that sensors and additional software 
adding some autonomy to telerobots are important. A basic example is to allow a mobile 
system to take itself from point A to B without operator input and then to request 
instructions. USBM needs guidance systems. Lasers are a possibility but will not work in 
undulating seams. Inertial systems, even using ring-laser gyros, probably lack sufficient 
accuracy over long cuts. USBM has been assisted by the NASA Lewis Research Center 
with IR and ultrasonic systems and are now examining vibrations in the mining machines 
and mine strata to determine whether the machine is in the coal seam. A sensor for 
tracking a coal seam would be an invaluable tool. Control research requirements begin 
with closed-loop control, then task planning for simple or reflexive machine control in a 
well-defined open area, and finally developemnt of strategies for mining more than one 
area. 

An important point that was emphasized for nuclear plant automation is that there 
will be only retrofit systems for some time to come. There are no current new orders for 
nuclear power plants, although EPRI would like to design automation into planned 
Advanced Lightwater Reactor systems. For this reason the Savannah River plant is 
emphasizing testing of such equipment as the Odetics walking machine which may be able 
to wend its way through a nuclear power plant maze. 

One interesting need is for disposable robots (expendable, low-cost) or disposable 
modules. Both the Army and Savannah River laboratories would like these futher 
developed. The Army is also interested in NASA developments of 1) computer-aided 
driving of remote vehicles, applying 3-D displays and point-to-point navigation updated 
every 30 seconds, 2) 2- and 3-D vision, 3) manipulators, and 4) low data rate 
communications. 

Few high tech improvements will be made soon to most earth handling equipment, 
said Mr. Leach, although they have simple remote operation packages and have announced 
a new unmanned electric forklift truck for warehousing. There are no plans to move 
toward camera-based type teleoperator systems. He also noted that with large equipment 
it is important to provide feedback other than such force feedback as that felt through the 
back pressure of hydraulic wheels. An example is audio feedback, which equipment 
operators have lost because of the evolution of, first, mufflers and then turbochargers and 
electronically controlled transmissions which eliminate the ability to calibrate the shifter 
with engine noise. 
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Ken Sebok asserted, on the other hand, that undersea operations are very similar to 
those in space and their problems may have mutual solutions. Vehicles operate in an 
essentially weightless environment without a source of ambient air and with sealing against 
pressure differentials required. Ocean activities must also consider problems of corrosion, 
high external pressures, ocean currents, low visibility and light attenuation. NASA could 
possibly assist with 1) sensors for acquiring a landing site, 2) sensors for alignment (e.g. 
valve replacement), 3) sensor monitors, 4) manipulators and end effectors, S) methods for 
achieving soft failures, 6) non-destructive inspection and test methods, 7) remote welding, 
S) vision systems to overcome lack of depth perception (stereo cameras and headsets are 
hard on operators), 9) inertial navigation, 10) stabilized platforms for better microwave 
uplinks from ocean surface to satellites, and 11) miniaturization of electrical and 
mechanical devices and systems. 

Dr. Brian Wilcox, head of a telerobotic research group at the Jet Propulsion 
Laboratory, provided the panel with some additional ’supply side" information for NASA 
technology transfer. Further discussion by -the panel was prompted by Dr. Wilcox’ 
descriptions of 1) light-weight manipulator arms for space applications whose construction 
and control paradigms could be of commercial interest, 2) the NASA commitment to 
force-synchronized dual arm control, to include handling of extended rigid objects, and 3) 
planning to include critical autonomous functions (such as on-board force sensing instead 
of force reflection) on space-based teleoperators controlled from the ground. This is 
essential to work around the 1 to 2 second time delay through the TDRSS satellite up- and 
down-links. 


CONCLUSION 

The discussions of motives and requirements for telerobotics application 
demonstrated that, in many cases, lack of progress was a result not of limited opportunities 
but of inadequate mechanisms and resources for promoting opportunities. Support for this 
conclusion came from Telerobotics, Inc., one of the few companies devoted primarily to 
telerobot systems. They have produced units for such diverse applications as nuclear 
fusion research, particle accelerators, cryogenics, firefighting, marine biology/undersea 
systems and nuclear mobile robotics. Mr. Flatau offered evidence that telerobotics 
research is only rarely supported by the private sector and that it often presents a difficult 
market. 

Questions on the mechanisms contained within the NASA technology transfer process 
for promoting commercial opportunities were fielded by Ray Gilbert and Tom Walters. A 
few points deserve emphasis: 


$ , NASA/industry technology transfer occurs in both directions and NASA 

recognizes the opportunity to learn a great deal from industry in the fields of 
automation and robotics 



Promotion of technology transfer projects takes a demand side approach, with 
requests to industry for specific problem identification. NASA then proposes 
possible solutions. 
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Commitment of motivated and technically qualified people on each end of a 
technology transfer is essential. 



NASA assures protection of proprietary interests and provides incentives such 
as exclusive licensing of NASA patents as part of the technology transfer 
process. 


! 

i 


NASA often enlists the assistance of other agencies (e.g.. Dr. Hodge mentioned 
the DoD Joint Technical Panel on Robotics), associations, or technical societies 
to recommend or participate in transfers with industry. 


The Office of Commercial Programs and its agents for technology transfer solicit the 
interest of industry and seek their specification/df problems or requirements with which 
NASA might assist. Prompt attention to anvjfjquiries may be obtained by contacting the 
Technology Applications Team, Research Triangle Park, NC (919) 541-6156. OCP 
gratefully acknowledges the opportunity/for the Commercial Users Panel afforded by the 
Jet Propulsion Laboratory and greatly appreciates the contributions of the panel experts. 
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Abstract 

W^dtteribe the design of a decentralized mechanism for al- 
locating assembly tasks in a multiple robot assembly work- 
station, Currently , <&r approach focuses on distributed al- 
location to explore its feasibility and its potential for adapt- 
ability to changing circumstances, rather than for optimizing 
throughput. Individual 4 gretdy* robots make their own local 


allocation decisions using both dynamic allocation policies 
which propagate through a network of allocation goals, and 


local static and dynamic constraints describing which robots 
are eligible for which assembly ta sk s. Global coherence is 
achieved by proper weighting of ‘allocation pressures * prop- 
agating through the assembly plan. Deadlock avoidance and 
synchronization is achieved using periodic reassessments of 
local allocation decisions, ageing of allocation goals, and 
short-term allocation locks on goals. 


1 Introduction 


The coordination of several robots in a flexible assembly 
workstation is a problem of growing importance. Three lev- 
els of coordination are necessary: 


1. Planning: Assembly tasks must be decomposed and 
represented as cooperative arrangements among sev- 
eral assembly robots [4]. 

2. Resource Allocation: Particular robots must be as- 
signed individual tasks in a detailed assembly plan, in 
ways that assure all tasks are carried out with optimal 
throughput [5j. 

3. Coordinated Motion Planning: Robot effectors must 
be controlled dynamically as they move close together 
in concert, while avoiding collisions. 


In this paper we are concerned only with the second coor- 
dination level - allocating tasks to robots. In a workstation 
of limited size, for tasks of limited complexity, we may be 
able to allocate robots to tasks using a centralized global 


allocation mechanism, for example, one based on heuristic 
search of the space of poarible allocations. As static task or 
workstation complexity increases, however, a global solution 
becomes more costly. Moreover, if we assume that orders to 
the flexible assembly station arrive randomly, necessitating 
reconfiguration, it will be very difficult to recalculate the 
global allocation plan each time to Told in" new orders with 
common subassemblies. 

For these reasons, we have decided to explore a decentralized 
approach to task allocation. In this scheme, a collection of 
robots b greedy for work; they are eager to take on whatever 
work they can do, and as much of it as possible. Each robot 
makes its own decision about what task to take on, based on 
its own local decision criteria. The robots’ greed is mediated 
by a set of dynamically changing allocation policies, which 
assures that global throughput requirements are met, and 
which encourages individual robots to assign themselves the 
most appropriate tasks given the global circumstances, such 
as order due dates, parts availability, etc. 

Our focus here (and our motivation for investigating concur- 
rency) b to explore the feasibility of a distributed solution 
and adaptability to changing circumstances , rather than op- 
timal throughput, or maximum production or reallocation 
speed. We have not yet addressed the temporal scheduling 
of task assignments to achieve desired throughput results, 
though we shall discuss some ideas for handling temporal 
constraints on processing. 


2 The Nature of Assembly 


Assembly is the process of composing higher-level structures 
from parts (primitive structures) and lower-level subassem- 
blies using assembly operations such as welding, fastening, 
screwing, riveting, inserting, etc. The assembly task allo- 
cation process must account for producing some number of 
copies of the high-level assemblies it generates - there is some 
lot of assemblies to make. Each lot is assembled on the basis 
of an assembly plan for an individual object in the lot, and 
on the basis of some due date for the entire lot. The due 
date places a time constraint on the assembly process. 
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Based on a preliminary analysis of several common manu- 
factured objects, we have derived a canonical form for the 
high level description of coordinated assembly tasks in an 
assembly plan. Figure 1 shows our scheme. An assembly 
plan is shown as a tree-structured composition of subassem- 
bly plans. Nodes in this tree are primitive assembly opera- 
tions. Assembly operations may require either single robots 
or groups of coordinated robots. We have only illustrated 
coordinated assembly tasks which require pairs of robots, 
but the scheme remains analogous for either single-robot 
tasks or those requiring more than two robots. 

Any high-level assembly A is composed of several subassem- 
blies, in this case S t and Sj. The assembly operation O s 
which assembles S t and S x to produce A comprises two 
subtasks, is, and Lsi, (which might be a HOLD and an 
INSERT operation, respectively). Each subtask has an 
associated resource constraint Rj^ and Rt», respectively. 
Resource constraints have static and dynamic components. 
The static resource constraints describe the machines which 
must be used to perform the subtask, and may indicate the 
type of tooling required, the lifting capacity or application 
force required, etc. Dynamic resource constraints describe 
constraints which vary with the allocation process (such as 
the real physical proximity of the required partner robot), 
once some robot has has taken on one task of the pair. The 
same description notation is used at each level of the assem- 
bly plan. Lowest-level subassemblies are made from parts 
(*>)• 

2.1 Allocation Pressure 

The existence of any unfulfilled task in the assembly plan 
creates two subgoals: a global goal of allocating some robot 
to the task and a local performance goal goal of perform- 
ing the task once it has been allocated. However, we use 
a decentralized approach to allocation, meaning that the 
individual decisions about which robots respond to which 
goals are taken by the robots individually, not by a global 
allocator. Still, to maintain global coherence, the global 
task-allocation goals must be ordered to reflect global pri- 
orities. We do this by establishing dynamic local policies 

to guide individual allocation decisions. These policies are 
not explicit decision rules; instead they are weighting fac- 
tors attached to each allocation goal indicating the global 
importance of the goal. The weights are called allocation 
pressures for the goals. A higher allocation pressure makes 
a given goal more attractive to any robot. The combination 
of allocation pressures, resource constraints, and the oppor- 
tunistic decisions of individual robots controb the allocation 
process over time. 

Allocation pressures attached to goals are dynamic and lo- 
cal. They change over time as some tasks are completed and 
others become more pressing. They are created by propa- 
gating allocation pressures through the assembly plan. 'Al- 
location pressures come from three sources, and are termed 
production pressure , coordination pressure , and consumption 
pressure . First, lower-level subassemblies must be created 
before higher-level ones can be assembled; this places a 


prece d ence on the order of assembly, and thus estab&b* 
some precedence for task allocation (e^, when these see 
fewer robots than assembly tasks, or some robots are faster, 
or better-suited for certain tasks than others). As each as- 
sembly order arrives, it carries a due date. The doe date far 
the completed lot provides the top-level production pressure 
which propagates downward (Le. toward finer-grained sub- 
assemblies) with increasing force. This serves to en courag e 
robots to choose lower in the assembly plan at first, 
it would make no sense to take on higher-level as s em bly 
tasks if there were nothing to assemble. As the doe dale 
approaches, the production premuxe mcreoscs, ageing the 
allocation goals over time. 

Second, if one robot (X) decides to assume some sobtid 
Li„ which is part of a coordinated task O t , it must be suae 
that it can induce some other robot (say, Y) to take on the 
corresponding subtask La, in order to proceed. We call tfa 
the principle of complementarity in allocation. As one robot 
makes a decision to assume a task, it creates 1) a new set ri 
dynamic resource constraints on the corresponding subtadk 
(because the allocation of one robot determines the physical 
location where the task will occur), and 2) creates coordina- 
tion pressure to encourage some available and appropriate 
robot to assume the complementary subtask (since without 
a partner, the operation is not possible). 

Third, the production of lower-level assemblies creates aa 
upward-propagating consumption pressure. This encourages 
some robots to take on the higher-level assembly tasks in or- 
der that work-in-process inventories (e.g. parts storage bins) 
and work flows remain balanced. Consumption pressure in- 
creases as more lower-level subassemblies are produced. It 
is initiated at the parts level, by arrival or availability a 
parts at the workstation. 

Allocation pressures are propagated using weighted prop- 
agation functions on the arcs of the assembly plan. The 
precise nature of the propagation functions is to be deter- 
mined by experimentation and theoretical analysis, whka 
await further research. 


2.2 Local Allocator Oecisioxunaking Cri- 
teria 

Individual robots make local allocation decisions by exam- 
ining the available unsatisfied allocation goals and selecting 
those with compatible resource constraints. From this set. 
the most highly-rated allocation goal is selected, and the 
robot allocates itself (see implementation section below). 
This allocation decisionmaking takes place with some fixed 
periodicity, as well as happening any time a robot becomes 
available for work. It is important to have repeated, periodic 
checks, for adaptive allocation and deadlock avoidance. 

If a robot has no current task (i.e. it is free to take on 
compatible task) then it may decide purely on the basis of 
compatibility. However, if the robot is engaged, yet is do- 
ing a periodic re-allocation check, it must also consider the 
changeover cost in deciding whether to take on a new task. 
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Changeover costs include the costs of retooling, opportunity 
^sts for not getting the new work done, and the prevailing 
performance goals for the task it is already performing. 

These decisionmaking criteria are purely local, both with re- 
spect to the individual tasks, and to the temporal unfolding 
of the global assembly work. It is possible that, for maxi- 
mum throughput, an individual robot should avoid taking 
on a task which is immediately available, and should wait 

for an upcoming but currently lower-rated task to which 
it is better suited. If the upcoming task is one in the cur- 
rent set of globally-known allocation goals, this requires each 
robot to incorporate in its decisionmaking criteria either l) 
meta-level control policies which can be constructed by some 
planner with a more global view or by integrating informa- 
tion about what other robots are doing by communicating 
with them [l], or 2) some predictive knowledge about the 
expected trajectory of the allocation pressures through the 
system. If the upcoming allocation goal will be generated 
by an assembly order which itself has not been generated, 
the individual robot will need even higher-level information 
about the likely arrival of new assembly orders. 

With concurrent access to all allocation goals in an assembly 
plan, there is a potential for deadlock. If two robots with 
conflicting reach constraints simultaneously choose comple- 
mentary subtasks, (e.g. L,. and La of task O*) the opera- 
tion cannot proceed, but there will be no way to deallocate 
any robot. Moreover, there is no criteria for deciding which 
robot should be deallocated. Thb is only a problem for 
subtasks of the same operation, because they must have si- 
multaneous allocation of appropriate resources. It b not a 
prob'em for subtasks of different operations, which can be 
allocated concurrently. Inappropriate allocation of robots 
across different operations will lead to inefficiency, but the 
adaptation mechanisms provided by the propagating allo- 
cation pressures, ageing of goals with respect to due dates, 
and the re-assessments of allocation decisions will force the 
system to correct itself. 

We use two mechanisms to prevent deadlock among subtask 
allocations. First, allocation access to all subtasks of a single 
assembly operation must be locked so that only one alloca- 
tor at a time can make an allocation decbion. After thb 
decbion b made, the new dynamic constraints posted on 
the complementary allocation goals assure that only appro- 
priate partners choose the complementary subtasks. Thb 
locking, while it reduces concurrency, does not create undue 
overhead, because the allocation decbions are made infre- 
quently, and are short by comparison to the amount of time 
it takes for actual assembly operations. 

Second, if no partner chooses the complementary subtask 
within the time constraint provided by the allocation re- 
assessment period, the allocation goal’s value may change, 
and the robot b free to deallocate itself and take on a more 
highly-rated goal (given the changeover cost - see below). 


3 Implementation Approach 

The allocation system described here his not yet been im- 
plemented, but here we present our design for implementa- 
tion. The implementation b planned for MACE [2^(, our 
concurrent Dbtributed AI testbed. The basic structure of 
the system b designed to be a global but dbtributed black- 
board system [3]. The blackboard itself contains the overall 
assembly plan and propagation links, and a collection of 
concurrently-executing allocation decisionmakers, one asso- 
ciated with each robot in the workstation. 


3.1 Representation of the Assembly Plan 

The assembly plan b represented as a collection of goab on a 
globally accessible but dbtributed blackboard. The black- 
board may be pre-segmented according to machine types 
and allocation constraints, to provide some efficiency in al- 
location and communication, and to allow for increased con- 
currency as individual allocators access different parts of the 
blackboard [3]. Each allocation goal in the assembly plan is 
represented as a collection of constraints on the type of robot 
which can assume the task. Each goal has two constraint 
sections: a dynamic and a static part. Constraints are 
descriptions of robot characteristics expressed in a flexible 
pattern-language [2|, to allow for partial matches, restricted 
matches, and variable bindings for the allocators. Static 
constraints are fixed parts of the assembly plan, whereas 
dynamic constraints are updated as allocation decbions axe 
made. The updating b done by a MACE computational 
agent which manages blackboard access. Allocation locks 
are implemented using the mailbox synchronization pro- 
vided by MACE, and are controlled by the blackboard level 
manager. 

Allocation goab are linked to one another with any of four 
types of urn-directional links. Each link comprises a type 
and a propagation function for propagating allocation pres- 
sures or constraints. The four types of links are production- 
pressure links , consumption-pressure links , coordination-pressure 
links , and constraint-propagation links . Constraint-propagation 
links connect subtasks of a single assembly operation, and 
describe how to update dynamic allocation constraints. 

3.2 Local Allocation Decision-Making Agents 

Local deebioninaking agents are individual MCAE agents 
which access the global blackboard using messages and demons. 
MACE provides a facility for remote demons, so that when 
a goal for which a particular tobot b particularly well-suited 
b posted or its evaluation changes, the robot can be noti- 
fied opportunistically. Direct access to allocation goab is 
achieved with messages. Local decisionmaking criteria are 
built into rules within each allocator agent. 
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4 Conclusions 


We have present td the design of a pc otnwiag theme Inc de- 
centralised allocation of tasks in a multiple robot assembly 
workstation. Further re s earch remains on the theoretical 
analysis of weights and feedback through the system. This 
analysis should include a control-theoretic analysis of the 
system, to show the weighting and constraint conditions 
under which it is possible to assure allocations will occur 
with msTwnmw throughput, and to indi ca t e the problem* 
atic coodtitions under which the system caa avoid thrashing, 
or at hut damp it. After implementation, we also expect 
experimental analysis to derive appropriate weights, reallo- 
cation periodicity, and adaptive performance behavior. 
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1„ ABSTRACT 

Modem telerobot control technology requires the integration of symbolic and non* 
symbolic programing techniques, different Models of parallel computations, and various 
programing paradt^as. This — paper rtesrrlhmyjfte Multigraph Architecture, which has b een 
developed for the implementation of Intelligentsia! -time control systems^^The layered 
architecture Includes specfffc computational models. Integrated execution ^environment and 
various high-level tools. A special feature of the architecture 1$ the tight coupling 
between the symbolic and non -symbolic computations. It supports not only a data interface, 
out also the integration of the control structures in a parallel computing environment. 




2. INTRODUCTION 

There is an ever-increasing demand for lag) roving/ the Information processing capabilities of robot con- 
trollers, measurement systems, or process control systems. The ultimate goal is to enhance system autonomy* 
adaptivity and functional performance. Since sqme of these features were previously provided by human 
operators, systems exhibiting these properties are/bften qualified to be "intelligent.” 

Essential extension of capabilities Is always based on the application of new techniques. Current trends 
in intelligent systems include: (l) the use cf parallel /distributed computing architectures, (2) the use of 
pare 1 lei /distributed programing models, an<r (3) the application of various artificial intelligence (Al) 
programing techniques. Since the new techniques typically are not substitutes but extensions of the conven- 
tional techniques. Integration has becom^a key factor in building intelligent systems. Symbolic and numerical 
computations, various programing paradfgms and different parallel co^mting models have to be merged in the 
frame of a possibly unified architecture. 

The critical system component /where most of the implementational problems of system Integration have to 
be solved, is the execution environment. The execution environment provides run-time support for the architec- 
ture, and couples various programming models to each other and to the underlying hardware system. 

This paper describes an experimental architecture - the Multi graph Architecture - and the corresponding 
execution environment developed for building integrated systems. After the summary of the background of this 
research, the design considerations are outlined. Then, the main components of the Multi graph Architecture are 
discussed, which is followed/by the summary of various applications and future plans. 


3. BACKGROUND 


A. Intelligent Systems 

There is a rapidly growing research interest in the application of Al techniques in robot controllers, 
measurement systems and process control systems. General, architectural issues of intelligent systems are 
analyzed in [1]. The generic architecture of intelligent systems ; s characterized by the introduction of the 
"know I edge -level ," which includes "knowledge-intensive* system ccxqponents providing high-level perception, 
modelling and planning functionalities. 


The structure and operation of the knowledge-level system components are typically model-driven. New 
possibilities offered by know ledge -based, model -driven automation in telerobotics are described in [2]. Ar- 
chitectural issues of model -driven instrumentation are discussed in [3] and the application of new techniques 
in the Know ledge -based Experiment Builder of a magnetic resonance imaging system is discussed in [4], The 
prospective of model-dri ven, know ledge -based systems in controllers is outlined in [5J. 

A common view regarding the structure of intelligent systems operating in real-time environment is that 
they mjst have layered architecture where "high-level," knowledge-based systea components synthesize, monitor 
and control the operation of the "low-level" sensory and processing activities. 


i 


i 
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B. Graph models of computations 

An important class of parallel computational Models are the graph Models. The coapvtatleoal graphs (or 
control graphs) are directed graphs, where nodes represent units of computations and arcs rep r esent dependency 
relationships. The general properties of graph Models are analyzed in [6]. and their classification 1$ given 
In [7]. In dataflow Models, arcs arise frow data depen d e n ces, and data are passed along the arcs In execution 
tine. In control-flow Models, not the data, but pointers to the data are carried; therefore, this Model re- 
quires the availability of shared memory. The computational units can be scheduled data-driven or deoMd- 
drlven. Data-driven scheduling Means that a unit Is executable If the necessary Input data are available. In 
demand-driven scheduling, only those nodes which are necessary to provide the requested data are activated. 

Graph Models have essential advantages in the context of intelligent real-tine systems. 

- Graph Models can uniformly describe parallel computations for different 
Multiprocessor architectures, such as distributed and shared Memory systems. 

- The granularity of the nodel can be "tuned” by selecting the size of the 
computational units. 

- The "Imperative" parts of the computation (l.e. the code of the computational 
units) are naturally separated from its logic structure represented by the 
control graph. This separation makes it possible to dynamically modify the 
computational structure. 

- The control graph can be easily represented in declarative form. The declarative 
representation is the key for using symbolic processing techniques to synthesize 
various computational structures, such as real-time signal processfng systems [4]. 

4. DESIGN CONSIDERATIONS 

The Hultigraph Architecture (HA) provides software framework for building intelligent systems in real- 
time. parallel computing environment. The main layers of the architecture are: the (1) Physical layer, (2) 
System layer. (3) Nodule layer and (4) Knowledge base layer. The basic properties of the Individual layers are 
summarized below. 


A. Physical layer 

Co^HJtationa? heterogeneity, various physical constraints {such as distance between computing nodes), and 
the typically high computation load require the support of different milt 1 pi e-processor configurations: 
tightly -coup led architectures with shared memory, loosely -coup led computer networks, and their combination. 
Special hardware components such as array processors or l/o devices might also belong to the hardware con- 
f igu rat ions. 


The primary function of the system layer is to provide access Mechanisms to the hardware resources. I« 
the current implementations of MA, the system layers are off-the-shelf operating systems, vhich facilitate 
services such as standard i/o, task management, intertask (interprocessor) communication and synchronization 
and real-time clock. An important requirement for the higher-levels Is flexibility to ensure the portability 
to different operating systems. 


C. Wodule layer 

One of the most critical requirements for HA is to support the synthesis and dynamic modification of 
various low-level computational structures (si9***l processing systems, control systems, etc.) in parallel 
computing environment. The key idea in the solution is the introduction of the module layer, which serves as 
an interface between the knowledge base layer and the system layer. The module layer has a special graph- 
oriented computational model, the Hu Iti graph Computational Nodel (NCN), which provides the following pos- 
sibilities: 


- high-level (possibly very high-level) declarative languages can be defined on 
the knowledge -based layer to represent various computational structures such as 
procedural networks, constraint networks, reasoning networks etc.; 

- these declarative forms can be interpreted and mapped Into a computation graph 
on the module layer; 

- the run-time support of HCH can schedule the elementary computations and “pass* them 
to the system layer for execution, taking advantage of the available parallelism of 
the co*>utational structures; 

- appropriate interpretation techniques can ensure the dynamic modification of the 
computation graph. 
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The MM "Motel* layer* suggests the trie* that this layer Is a "Module library* consisting of typically 
saall prograa Modules written in C, Fortran. LISP etc. These nodules are structured to for* a conplete program 
by the definition of the amputation graph. 

D. Knowledge base layer 

High-level, symbolic amputations are laplenented on the knowledge base layer. Althou*! the actual struc- 
ture of the knowledge-based system components are strongly application dependent, the parallel computing 
environment and the features of the underlying module layer sake the elaboration of a generic programming 
model desirable. The main purposes of the M*-level programming model are: 

- to support the structurization of the knowledge-based operations Into concurrent 
activities. 

- to facilitate a standardized, high-level communication system among the activities, and 

- to provide Interface to the nodule layer and HOI. 


A strict requirement Is that these services have to be implemented as extensions to one of the standard 
LISP systems in order to preserve the compatibility with different AI toolsets. 


5. OVERVIEW OF T* MULT I GRAPH AKH1TECTWUE 

The basic costing models used on the different layers of HA and their relationships are represented In 
Figure l. 



FIGURE 1. Layers of the Multi graph Architecture 
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A. Autjggm ous Co— uni eating Objects (ACO ) 

The basic system structurization principle on the knowledge base layer is provided by the concept of 
Mow— C—ri eating Objects* ACO is a straightforward extension of the "object" concept of object- 
oriented languages such as Flavor [8] in the following sense: 

- ACO's are fully autono— is system that can run virtually or physically parallel, 

- ACO** can be dynamically allocated and can conpete for the sane resources, 

- they co— inicate with each other by neans of a fully asynchronous co— unication 
protocol. 


The naln purpose of ACO's is to provide a standardized "object shell" around a variety of heterogeneous 
knowledge-based system components. The co— ml cat Ion "methods” are standard elements of the object shell, and 
hide the details of an actual Implementation from the application program ers. 

Various object types have been developed for supporting specific applications. These objects typically 
Include a "knowledge base," wffich is represented by a special representation language. Some of these object 
types, such as Procedural network Object (PNO) and Rule Hetwort Object (RNO) are described In [9]. 

B. ttiltl graph Computational Model (HOI) 

The different object -types are facilitated with an appropriate Interpreter or Incremental compiler, which 
maps the actual knowledge base Into a computation graph on the module layer [9]. While ACO's serve as a sym- 
bolic representation and Interface to possibly complex functional components of the system (e.g.» a signal 
processing system, rule-based system, associative database system, etc.), the computation graph on the module 
layer constitutes their actual execution environment. This relationship between the ACO's and their execution 
environment has the following advantages: 

- Execution of the operations represented by ACO's occurs In a parallel execution 
environment offered by NCH. 

- The Interpreter (or incremental compiler) "methods" of ACO's, which build the 
amputation graph, can dynamically modify the graph, as a response to an external 
message, or to a feedback from the execution environment (a mechanism for 
implementing "self- modifying" signal processing systems is described In [4]). 

- The system fully Integrates two different parallel computing models. ACO’s form 
the "macro-structure" of the system, and they communicate by using the services of 
loosely coupled distributed systems (typically message passing). The computation 
graphs provide the "micro-structure" of the system components. MCH efficiently 
supports medium-level (subroutine size) computational granularity, and can take 
advantage of tightly -coup led multiprocessor architectures with shared memory. 


C. System tasks 

The computational model on the system layer is provided by the actual operating system. The key concept 
is the "system task," which represents a "slice" from the processing capacity, and can access to various 
resources. The elementary computation units that are scheduled by the run-time support of MCH are executed by 
the system tasks. 

6. HULTIGRATH COMPUTATIONAL NOOEL 

MCM can be characterized as a control-flow model. The control structures of computations are represented 
by bipartite graphs that are built of actomodes, datanodes and connection specifications (see Figure 2). 

The actomodes are associated with the elementary computational units, called scripts, which can be 
written either in LISP or in any other language, such as C, Fortran, Pascal, etc. The scripts do not know 
about their position in the control graph: they communicate with other graph components through the 
input/output ports of the actomodes. The actomodes are associated with a local datastructure, called 
context, which can be accessed by the script* If the code of the script is reentrant, it can be attached to 
several actomodes. In different computation problems the scripts may be quite different: a script may be an 
interrupt -driven l/o handler, a transformation of the input data arriving to the actornode, or an Interpreter 
module, which Interprets the symbolic form stored in the context of the actornode. 

Datanodes store and pass the data generated by actomodes. They can be either streams with wltiple 
output ports, or scalars. The streams maintain the partial sequence order of the data generated during the 
computations, irfiich preserves the overall consistency. 

The control graph can be operated in data-driven or in demand-driven mode, or in a combination of the two 
modes. In data-driven mode, the data sent to a datanode propagate a "control token" to the connected actor- 
nodes. The actomodes will fire according to the specified control discipline: in ifall mode, at least one 
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FIGURE 2. Conponents of the Multi graph Computation Model 


control token wst be sent to all of the Inputs of the actomode: In Ifany mode, every received control token 
will cause firing. In demand-driven mode, the request for data sent to a datanode will generate a control 
token If the datanode Is “eppty." This control token will fire all of the actomodes that are potentially able 
to provide the requested data. The demand propagates backward along the control graph until data Is generated. 
From this point a forward propagation starts, which finally provides the requested data. A more detailed 
description of the computational model can be found In [XO, XI]. 

The mn-tlme support for the HCM Is provided by the Multi graph Kernel (*C). The structure of the MC Is 
shown In Figure 3. The control graph Is represented In the descriptors, tdilch are manipulated by various 
kernel functions. The Control Interface functions are used for dynamically building and modifying the control 
graph. These functions are Imbedded In a LISP system, where the various graph-builder Interpreters and In- 
cremental compilers are Implemented. The Nodule Interface Includes the data/demand propagation kernel calls 
for the scripts. An Important feature of the system Is that actomodes with scripts written In different 
languages can be mixed in the same control graph. (The necessary transfer routines are invisible to the user.) 
This feature Is used for creating tight coupling between symbolic and non -symbol 1c computations. Ti^it cou- 
pling means that not only data structures can be passed between the two kinds of computations, but there may 
be a fully integrated control structure. 


CreateActor( ): CreateOata( ): 



FIGURE 3. Structure of the Multigraph Kernel 
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The most complex part of MC Is the System Interface. The System Interface schedules the elementary com- 
potations that are defined during the data /demand propagation. The conputatlonal units (the scripts of the 
fired actomodes) are passed to the available systen tasks for execution. The mlr— rut n ecfc a n lsa of the 
Systeu Interface ensures that subsets of the control graph can be <frna*tcally associated with one or more 
systen tasks that Include the necessary resources for executing the scripts. This mechanism provides a very 
straightforward way for dynamic resource aanagenent In nultlprocessor configurations. 

Two I^KHtant Inpleuentatlon Issues are the granularity and the memory aodel. The lower Unit for the 
reasonable conputatlonal granularity Is basically determined by the overhead of I9C. Since MC currently Is 
1^>1enented In software, the overhead Is Introduced by the control token propagation functions. On the 68000 
processor-based IBM 9000 system (clock frequency Is Miz), the overhead Is about 800 microseconds: on the VAX 
78S Implementation Is less then 200 microseconds. Due to the construction of the NX. the overhead Is basically 
Independent from the size of the control graph. 

Since PCM Is a control -graph model where the pointers to data structures rather than the data are passed 
along the graph, the model requires the presence of shared memory for those tasks (and processors) that are 
assigned to the same subgraph as execution resource (see Figure 4). In order to provide flexibility toward 
architectures which do not support shared memory access for the processors (hypercube architectures or dis- 
tributed computer configurations), a simple mechanism Is Implemented to link control graphs that are allocated 
In the local memory of the separate nodes. The scripts of receiver and tr a ns mitt er actomodes provide a logi- 
cal link between the separated subgraphs and Implement the data transfer fay using the actual services of the 
underlying system layer (e.g. , the message passing services of DECNET in the VMS/DECMET implementation). At 
these links, the control-graph model Is "transformed - to dataflow model, since the actual dat as t ruct u res - and 
not just pointers - are passed to the "remote" nodes. 

This method makes It possible to generate large processing networks from their symbolic representation In 
distributed computing environment [12]. 


NODE -1 WE -2 



FIGURE 4. Memory Model of MCM 


7. STRUCTURE OF THE EXECUTION ENVIRONMENT 

The simplified structure of the execution environment supporting MA can be seen In Figure 5. MX is imple- 
mented as an additional layer to a standard operating system. Depending on the computer architecture and on 
the features of the particular operating system, I* may exist in one or more copies. The System Interface of 
MX is a well structured, modular program which makes porting the kernel relatively easy, even to devastatingly 
different operating systems and real-time supervisors. 

The Module Interface functions of MX can be Invoked from LISP as well as from other languages. This 
ensures that scripts can be written In different languages, and existing module libraries can be easily inter- 
faced to liC. The Control Interface of MX is imbedded in LISP since currently we use LISP as implementation 
language of the knowledge base layer. 

Various high-level software components such as the generic object shell for ACO’s and the standard 
methods of different ACO types are implemented in LISP. 

For distributed computer configurations, the LISP system (in the first Implementation FRANZ LISP, 
recently changed to Zamon Lisp) has been expanded with the Comnunicating LISP System facility, which provides 
task management and asynchronous message passing primitives [13]. 
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FIGURE 5. Structure of the Execution Environment 


8. EXPERIENCES 

MA has been i«*>lemented on very different computer configurations and has been used for various applica- 
tions. 

An Intelligent Test Integration System (HIS) has been implemented in the Space Station laboratory of the 
Boeing Aerospace Company In Huntsville* AL [12]. The purpose of ITIS is to support the automatic generation of 
test systems in real-time* distributed co*>uting environment. ITIS is implemented as a knowledge base layer 
above the conventional test system components, and can build co^>lex test configurations from the symbolic 
specification of test scenarios. The computing environment is a VAX network with the VMS /BECHET operating 
system expanded with special hardware units. 

A different application of the layered HA architecture is the Knowledge-based Experiment Builder (KEB). 
which was developed for the M.I.T/IBM experimental HR I (Magnetic Resonance Imaging) system [4]. The core of 
the KEB is a high-level representation language for signal processing schemes and a smart interpreter* trfiich 
can generate the appropriate version of the real-time signal processing system* and which is able to recon- 
figurate it for specific events. The computing environment of this system is the IBM 9000 computer with the 
CSOS real-time operating system. 

Various computational structures have been developed and are being investigated for tne HCM, such as a 
hierarchical planner [14], knowledge-based similation builder [15], pattern-driven inference system [16], etc. 

9. CONCLUSIONS AND FUTURE PLANS 

The integration of symbolic and conventional programing techniques, parallel computing models of dif- 
ferent granularity, and various programming paradigms are essential conditions for the successful implementa- 
tion of intelligent real-time systems. The Multigraph Architecture has proven to be a good approach to solve 
the problems of integration. It provides a generic framework, programming models for stnjcturizing software 
components and various tools for the actual implementation. 

We have practical experiences with implementing systems in single -process or (IBM-AT/MS-DOS), single- 
processor multitasking (VAX/VMS and IBM 9000/CS0S) and distributed (VAX network YMS/0ECJ4ET) computing environ- 
ments. As a next step, we intend to implement the execution environment for tightly-coupled mil ti processor 
configuration. 

The system offers a convenient method to describe and implement "sel f -modi fying“ signal processing sys- 
tems. Further research is needed to utilize this capability in tne design of structurally adaptive measurement 
control systems. 
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t. ABSTRACT 

Supervisory control systems must deal with various types of hmeSgence' 1 distributed twoughout the layers of 
control Typical layers are realtime servo control, off line planning and reasoning subsystems and finafly, the human 
operator. Design methodologies must account for the fact that the majority of the intefigence wH reside with the human 
operator. This pgrterforeffmopjhietaichica feedback loops as conceptual boating blocks that provide a 

common ground for man-mactthe fcteractfon^Kamptes of types of parafetism and parallel implementation on several 
f comp u ter architectore are ateo dfecussetfcv /, 

A ;/ /•,, , 

2HHfROOUCnON 


Designing systems that involve a V>t of interaction with humans is a cSfficutt task. Although humans are intefigent 
foey are also unpredfctabfe. ft is easier to conceive of completely 'autonomous* systems, requiring little man-machine 
interaction. But with such systems human operators run the risk of losing control and at some point the operator may 
want to redked assist, troubleshpot or enhance the system. Thus, it is desirable to be operating somewhere in the 
middte ground, where systems am designed to be autonomous only in the sense that they are part of a larger system that 
provides a means of imeractfon *wh humans. The concept of "supervisory control" provides such a framework. 

2.1 General System DesJpi Goal 

Put in simple termsTrt is desirable to have systems that are refiable. flextole and exportable (Figure 1). Refiabifity 
implies that the system /arely fails and that there is a quick fix for faflures such as plugging in a new readly available 
module. Along with pqor testing and evaluation is the development of run-time aids such as fault detection, isolation and 
ctiagnosts systems, ay of which improve retiability. A flexible system is one that can perform many different functions or 
tasks and can operay in cfiverse environments. Such a system must consider many alternatives and be able to “understand - 
its environment After a system is fielded it is invariably discovered that new capabilities are desired or required. An 
ex p a n da ble system /can easily incorporate new functions and more knowledge. 


Flexibility 

- Variety of Task* 

- Variety of Environments 


Reliability 

— Few Failures 

- Easy Repair 


System: 



Manipulators 
and Sensors 



- Communications j 

- Life Support 

- Power 

- Thermal 



Expandability 

- New Capabilities 

- New Knowledge 


\ 

r 


Figure 1. Design concepts 
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Many of these design goals boi down to provkfing fw system with "Emergen ce *. Although twe has been much 
progress in the development of systems that deplay Imited forms of totetegencM « often the casa fiat urtien inteBgance 
is required human operators must be i ncorporated. Therefore, toe development of complex systems now and in tie near 
future wfl rely heavily on human operators (Figure 2). 




Figure 2. A definition of ** supervisory control" 


2.2 Supervisory Control 

To effectively operate a system, a hunan must focus attention and can only expend a certain amount of energy. The 
more autonomous toe system, the less attention required from the human operator. If the goal is to have less and less tor 
the human operator to do the human would ultimately become a " w atchm a n* aslee p until an alarm sounds. Unfort u nately, 
human expertise should be used more efficiently. Thus, the proper dfcection to move in is that as a 
particular system becomes "smarter*, the operator devotes less attention to ft and devotes increasingly more attention to 
a second system, and so on. From this perspective, the operator has various, possfcfy autonomous, subsystems under his 
coned as he orchestrates the performance of the overal system. 


2.3 Amplifiers of Human Capabilities 

Machines, in general, enhance human capabOties. Machines give stength. reach and dexterity not normatiy 
possessed by humans; they can protect presence into areas that would not normatiy be entered; they allow detection of 
things that are beyond the capabity of human senses and foey can enhance our analytical atofity by rapktiy analyzing data 
and summarizing the results (Figure 3). In this sense machines act as •amplifiers’' of our capacities, inducting the 
amptification of our intelligence. From this point of view, the design of a complex system is seen in terms of a cantrafly 
placed human operator - the soiree of in te teg o n co - supported by a dvanced man-machine rntartaces teat provide an array 
of "tools*. The tools are not intended to replace man. but to make it easier tor him to perform dftficult tasks. 
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Figure 3. The oysters acts as an "anpl if ier" of hunan capabilities 


3. LEVELS AND LOOPS 

In seeking a common ground for both man and machine, two design techniques stand out hierarchical decompositions 
ano feedback loops (Figure 4). Although these two concepts are not mutuafy exclusive, they have contrasting properties, 
emphasizing different aspects of the system under scrutiny. Just about every system has elements of both. In tact the 
same system might be described in terms of a hierarchy or a particular feedback loop, depending on the emphasis desired. 


HO 







First we w* discuss hierarchies. 



Figure 4. System design oethodologies-levels and loops 

3.1 Hierarchies 

Hierarchies are pervasive, appearing in societal structures such as tie mStary, corporations, churches, and in 
more absfract areas such as general problem solving (Figure 5). Some of toe simplest hierarchies appear in knowledge 
representation, resulting from natural taxonomies such as "part-of". Is a" or "kind-or. Similarly, spatial 
decompositions such as topogr a phical maps or geometric descriptions of a robot environment treated at various levels of 
resolution provide examples of hierarchical decomposition. Approaches to complex problems such as speech and image 
understanding invariably involve levels, with the lowest ones associated with “primitives', the mkfctfe ones with various 
patterns and the highest ones wito symbolic int e rpretations. In [1], the organization of functions in a robotic environment 
such as an automated factory is described as a hierarchy, with high levels related to planning and reasorxng about the task 
and low levels identified with servo control. 



Figure 5. Examples of hierarchies 


After looking at several hierarchies some general properties emerge. In more complicated scenarios, such as the 
decompostion of tasks for a robot, the highest levels are associated with reasoning and planning typcafy using very 
general global or abstract knowledge. As a result the highest levels tend to ieiy on thoughtful, slow, symboic processing 
(Figure 6). The overal goals of the system are set at toe highest levels, typically considering many alternatives, 
representing toe "goal-driver:' aspects of the system. In contrast, toe lowest levels are associated with 'action', using 
specific local knowledge. At toe lowest levels are numeric, algootomic processing that results in fast reflex action 
representing the "data-driven" aspects of the system. The overal hierarchy usuaty f* presents a decomposition of 
tactions toat is equivalent to toe successive reduction of a problem into smaler and easier subproblems. 

The dynamics in a hierarchy become a pp ar ent when it is resized toat information can flow in either direction, 
creating loops toat can s trade** several levels (Rgure 7). Furthermore, each level must be able to communicate with 
levels above and below, poss&y requiring separate, local, languages, and also possbiy swamping the middte levels with too 
many messages. 


A hierarchical approach is not ateays easy to apply. For complex systems, defining the appropriate levels is quite 

141 


















c»np fc^ fiwiing^ao^na^samedWta^Modated^ o »>^pro b >«m»o^ «ndfcn o^dg>i» praa <nafa n . 

7I» probten reduction inharant In ■» NaranNcal deewnpcsilton cm ta> alwrt bacauw ol comptai ittanalaltont among 
ttbpnMn. N ava rlh al aw . a hfen reWca l approach % axtramefr powartU in is atoMy lo fluid* fia dasign o< » compto 



• Th* iMh Ait Eacy T« (Mm far T* 


• For Cw a plw S iMM tinwt . IN Art Up Apart 


If Latafc Aio Mat Do* — ■» -, .. <tM— ^lurtwidit 

Figure 8. The challenge of hierarchies 


Whit toe use of hier a r ch ie s refleas the need to decompose large systems into smafler tactable subsystems, 
feedback reflects the need to account lor uncertainty. » a human operator or an autonomous systom had perfect knowledge 
toere work) be no rued for feedback since flings woUtiafcvays go acoordtog to plan. Unfortunately, knowledge is often 
i nc omp le te and erroneous andasaresuil actons must be Mowed by some form of checking. 

The basic process underlying a feedback loop consists of two steps, repeated unfli euccessM : 1) using feedback, 
eHher trom sensors or otoer sources ol intocmatfon, compare current state wflh desirad stale; 2) based on the rasufts ci 
f* first step, decide on an action toatwte move the system closer to toe desired stats. Figure 9 depicts several types of 
toeaoacxpops. 



• Fwdtect HaKn Accmm for UwntiiMv 


• Tht EmpMM K om iNotMtnM Aay w T w mmd a Goal 

Figure 9. Feedback loops 


The test step assumss that sutecfent infor ma tion can be gathsred fcom toe particular domain of interest ( ie. NvortcT) 
and represented in a form that can be compared unto some representafion ol toe goal For example, a simple sera 
confol loop machea a desired with a measured faiectory to creete an error signal ter toe 'coneofler' which attempts 
to drive toe error to zero by the proper actuation. In tots way a rtfeflve* simple modal of toe NvorkT can be very 
effectors. By focusing on toe crifical i nf or ma tio n as represented by toe error signal, a rapid response can be easfly 
formulated: but robust behavior (such as a dapt a tio n to toad changes, coftsfen a wadance and compflantconeol) is dMcufi 
to achieve without detaflad dyrvavnic modsfs. That is, in most cases toa ostision on what to do at aach iteration is not 
obvious. Altoough toe iterative rmiae of f» feedback loop impies incremental or local progress toward toe goal. global 
knowtodgt must be used as wel to nod dead ends and local externa. 

Anotoer basic example fetoelmeans^rids'anteysfeusedasaproblamsokfingperackgm. Here, toe r e pre s entati ons 
of stales, goals and available operators can be quftt absfact bu toa approach can be described in terms ol sloop. A 
comparison of toe current with the desired stole detects a dWerence fiat is used to select oper a tor s that can reduce toe 
deference. Typically, this produces a search process as various operators are toed out on a simulated •raricT. Once 
again, though, one of the major dffcuftes in applying such a procedure is the need to reoognae global contamts at toe 
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Figure 10. Typical task 




Figure 11. Search 


4.1 Search 

Unless tie environment m extremely con st rained. ee ar chtoQ tor an object wii rely pnmarifr on vwi ( a* posstoly 
range image* ). This sensory data must bs processed end co mpared with inland m od e* of tie objects physical 
properties. taction* aspects and expected location. In gener*. tm is known to be a very cfiflcut task, involving 
hierarchical sauctures tftat attempt to organize knowledg e chwen processes arteng torn e xp ectations. and tie data 
driven processes vising tan tie reduction of sensory data into image primitives such as sdgaa. Ines and surfaces. 
Searching lor objects wto revere a significant amount of he*> torn tm human supervisor. 


Navigation entafe tie interplay of an interesting combination of information :l) pnor knowledge ol toe environment 
typacatiy desafeed in global terms since tie delate would be had to keep track ol ( tor example, a des cri ptio n of tie 
stucturef componen t su^ias tie main passa g e w ays and knoem beacons could be av a tiatl a. bit smatiofaeadaseoidd have 
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tobodtocovered):?) The robot* sensors such as vision and prorimiy would have to provkto toe local biloimaion about 
toe envkonmsr* and useitoavold obstacles an d t aoognizaqnrtaarfcs; 3) Mam* moion sensing coukl be used to gjw toe 
robot an asHmsSs of to position independent of too stomal sensors; 4) Italy. planning paths tost nogototo best 


la onto to suc cassMy navigate. N robot must use s» 
spots and dead ends vtols taasoning about progress 
ia a vary sametursd environment. but in raal 


m niDniHBQn, 

too goal. 

toe problem is quite 


inquiring tool 



p lxure 13. Manipulate 


43 Manlpuiata 

H me robot is success** in gating to toe object (Figuro 13). it must, among other Mngs: d a tarmlna vtostoar 
neighboring objects must be moved to get at it. inspect toe objact to see 1 1 q dataettva. detect toe mad lor spatial 
propping tools. And grasp points, and essmate toe weight and teengto of toe object Altar graspmg toe object aanaoro 
must monitor ovorol stabity a n d detect si ppaga a nd overload. Once toe object it sears* in toe robot* grasp, toe robot 
must navigate to toe goal as a dtoerent dynamic system etoicb may ator toe chosen routs. The majority ol Usee 
subtasks a* require toe intervenion ol a human operator. 

4.4 Complexity 

Even a cursory view of toe go-Mch example iml carot Hat toere to a tremendous co mpto xi ty involved. Upon 
analysis, toe vaious subtasks are interrsl alert, and toe idenMcalon of various sauctoroe such as hierarchies and 
feedback loops can only have Med eaeet since even toey are tangled among each otosr. O eani ng •primtove' tasks, euch 
as peg-in-hoie and tom-crank, q an exce i en t way to control toe comptoaqr. AMtough toto approach a somewhat baaed 
by toe dtocisle primitives and combinatorial erotoatoq. > r epr ese n ts a sold hierarchical approach tout can buM up 
complex tasks bom easiy understood prMfees ( see P) tor an roamplo of Ms approach ). 

R q not uncommon lor a robotics engineer to graduaiy resize toat what at tost appeared to bo easy to autosMte q 
extsmeiy complex, defying ai attempts at auto m ation. The history of vamn and natoral language research provide 
classic examples, but toq phenomenon nms toroughout toe rotates world. 

Humans have an uncanny abiity to sknpMy toings. The environment presents an exsemeiy complex array of 
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Figure 15. Complexity 


4. PARALLELISM M SUPERVGORY CONTROL SYSTEMS 

ResHirra execution fdbtftoutodsupervtoory control sysromswfisqfw imptomenl f ion on piraflf and advanced 
computer arch ite ctu res. This requirement is driven by the types of processing and processing requirements in 
supervisory control systems. As • the case with iroderstanfng and descri bin g a problem, it is useful here to tank in 
twins of levels or hier ar chie s . Since toe types of pro c e ssi ng in a tu psnsao r y co n sol s y st e m are so diverse, iritonafte 
imptemen ia lton wa require a network of spoof purpose, pa ra Sa l , and serial a r ch ite ct ur es . In toe remainder of tois 
taction we briefly consider toe types of proc es sing to a supervisory control system and toe suitabiCty of various 
archSffctjros to these levels. Furthermore, we suggest that the b igge st p roblem teeing per s is t impiementfion of 
supervisory control systems is lack of mature software tools end techniques lor implsmsntotio n of these sysitms on 
dfributod and p a r a ll el ar ch flec lu ro s . 

5.1 Processing Requireme n t s 

The processing requirement of a s^ervtoory control system can be roughly dfcidsd into twee levels which suggest 
toe types of arontoctures which are beneficifL These levels are not neceesarfly musiafly exclusive as tasks may occur 
which straddle several levels. 


5.1.1 Low Level Proc essi ng 

This level is is connected wito infraction wito toe environment through sensors and commends to effectors and 
actuators. Typicf tasks performed M tod Isvf are: sensor dwVocosstogffitohng such as aega feature emetion 
(convolutions), associalive retoevf. and simple trajectory generation. Thto levf to genecafly characterized by toe 
toiowing requirements: 1) High ba ndwid to VO rases; 2) Numeric pro c e ssi n g ; 3) Algorithmi c processing ratoer than 
search among atematos; 4) IdenflcaL repealed operators on bounded regutar data stuctures. 


Specif purpose hardware is often appfied to much of toe processing f tois toroL The spsdf purpose hardware can 
be serial or parafleL but typicafly is designed tor very specific functions such as comcWons and does not support ganerf 
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$•40 processors |4J •*• perticufariy wefl suited to toto lavel of processing. Examples of Ms class of paraM 
proceaaor are toe Mac IV and MPPpj. These machines are particutariy weti suiad tor problems involving identical 
operation s on bounded tegular data structures each as matrix mutifpticatiort. peraM sorting. and fast louder tanstom*. 

MU) processors HI such as Vie B8N Butterfly and Cm* have atoo been appled to toese types of problems, however, 
toey have not achieved toe same level of p erformanc e as SIMP ma chine s p^. 

Hi MM Level Processing 

I me mm concerns more general purpose procesiisiQ man emier me lower or rearer eves, in* wvo gmevf 
perform s operations on M M re d, preproce ss ed sensory data fcom tie tower JeveL Typical fcnctions performed hern are 
vuiysis {oopKX Sfnpw wnio® rh™ wo 0 (XnttNon pfooiMis usng wcnw^iw woi 

aady na m fc p rogramming. This level is characterized by toe Mowing requiremente: 1) Medfcm bandwfcfth (Orates; 2) 
Varied operetion e on regular or irregular data structures co n t ai ning simiar data types; 3) Optimized choice among a 
bounded aatot alternatives. 

SttP ma chines can be effe cti v e ly apptied to this level ol p roces si ng ti toedtea lo be oper a ted on can beaffocated 
untiormly across tie processing dem ents and tot processing performed on toe data el em e n ts is simiar. In the absence of 
toese. many of the eie m ents in me processing array re mai n kfle bo much of tie time resulting in inefficient use of the 
system. In such cases. HMD processor can be appled. 

The MfftiD. or Multiprocessor architectures, are the most general paralel processors and are capable of executing 
muftfpto corttof threads in parafleL Tightly coupled (shared memory) MIMO computers era applicable when the problem 
requires a high degree of inte r pro ce ss communication and a high degree of sharing of objects/tesources in the system. 
These mechiries can supports tner degree of pmcess granularity due to me high inltrpto ce sscommurti ca tion ba ncKridm (on 
the order of memory bandwidth). If tha problem cals for minimal data/resource sharing, then loosely coupled 
mutfprooesaors are appticable and mi not be subject to performance degradation due to resource contention as is often 
me case in a shared memory system. 

5.1.3 Mgh Level Processing 

Typical functions performed at mis level are route and path planning by heuristic search such as A*, mission 
planning, logical inferendng. and object recognition and understandtog. This level operates on data that has been 
preprocessed by each of the lower levels and typicatiy controls the lower levels. It is characterized by: 1) Low bandwidth 
i/O rates: 2) Varied operations on afferent data types and irregular data structures; 3) Symbolic processing: 4) 
Heuristic search of a large number of alternatives. 

MMO machines are the most widely appfied architectures to this level of processing. This is due to the flexttity in 
these systems since afferent processors can be appied to different aspecte of the problem. Examples are paraM 
branch-andbound search and paralel execution of logic programs pj. 
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5.4 ParaM Programming 

The biggest chalenge facing me use of paraM processors tor supervisory control systems is the complexity of 
programmng many paraM machines. This cftfficufty arises because programmers must often famffarize themselves wtih 
tow level details about the paraM architecture as weti as paraM progr am ming techniques before they can become 
effective users of a paraM machine. In order to take advantage of the paraM hardware, conventional programs must 
often be r a mit ten and embedded wtih system cats tor memory and process atiocation. In addtion. Mon software is ported 
from one paraM machine to another, it must again be lewritien to account tor a afferent a r ch itect u re and set of paraM 
language features. ParaM p rogram m ing technology is maturing, and advances in me totiowing areas wte resut in more 
efficient programming: 


1) Opti m izing compilers for serial languages. Such compaeis w* atiow tor portabffty of existing codes to paraM 
machines by making the underlying paraM architecture transparent to the user. Furthermore, this wiff allow for 
portabtity of software across different paraM architectures. Progress in this area has been made tor both conventional 
languages such as C and Fortran [6] as wea as Ai languages such as functional languages and Prolog {9). We are axrertiy 
working on a paraM interpreter for Prolog execution on toe Butterfly which automaticafly optimizes toese program s lor 
paraM execution. 

2) A common set of paraM programming and language afastactfons which can be appied to toe wtous languages 
and ported across different paraM machines. This coupled witi optimizing compilers aflows tor the use of p a rt ial 
archftectjres at multiple levels of expertise. Novices can write progr ams which are automaticaly paraHefezed and 
optimized tor the particular architecture. As they become more experienced and knowledgeable about toe particular 
a r c h it ect ure being used, they can optimize programs even further by embedcflnp toese abseactions - or compfler directives 
* within toe* programs. 

3) A common set of resource sharing and process intercommunication protocols which w 41 atiow for programs or 
processes operating on heterogeneous processors in a network to communicate. 
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6.C0NCLU90N 

There am many open issues concern^ tie development of supervisory systems for Space Staton btftw key taor 

to tw man-machine fraarphy. System daeign maSwdotagiei mutt dtowlht humans who pa n— tie mejorky of toe 
inteflgence to use tie subsystems as toofc. The tubryrtwnt shoUtfwnptoy hu n wn c apafatttoe by dbwfogfie operate to 
dkectkoUbtoshoot end s tt a no s system pertownenceL F^s conceptual stBndpojntbotiliieiaicttctoaeooiitooseiofs and 
feedback bops see common to man arM machine fcarebyprovtfng a simple ftrameeroik tor interaction. Unloriunttdy. 
typical tosio are quite complex, requiring twt humans eokwtiem. As experience is gdned.te human operators** be ii 
the postoon to encode new procedures, prodded Ml toy have tie right development tools. 


ParaMimplemertatonof toese supervisory control systems wff provide the performance necessary to sustain 
ree l S me ex ecuton of such s yst e ms . I d ea B y. dtot rto utod computer systems co n sisti ng of serial, special purpose, and 
parsed processors waste* for optimal p erfor ma nc e ol tie dMersnt processing requirements of such systems. The 
major dkfcuty. h owe v e r , in using pafiiel aroftitsctes e ff e cti vely Is toe complexfty of p r o gra m ming them. Recant and 
tortcoming advances in paratel software technology wfl reduce tie impact of tits problem and alow for paraAal 
implemenlatfon of tiese systems. 
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Abstract Process management combines complementary approaches of heuristic ramming and 
analytical p ro cam control. Maa agu nnt of a continuous p roe m r eq uires moniioring the 
environment and the controlled sys tem, smearing the ongoing situation, developing and revising 
planned action^ and continuing the ense nti on of the actions. For knowladg eintea e iv e domains, 
p ro cam m a nagemen t entails the potentially time-frame rl coope ra tion among a variety of expert 
ajneum . By r e d e s ignin g a blackboard control architecture in an obyct-oricated framework, we 
obtain an approach to proc e ss nunigtmrnf that con si d era bly extends blackboard control 
mechanisms and o verc om es limitatione of blackboard systems. 


1. Introduction 

Many f mare military and apnea applications wiU impure con- 
trol of eatonomoiie or semiautonamooe systems operating in 
dynamic environments. ImMm Am n r ffrtr control is tbs 
teak of ap ply ing knowledge intensive r saso ning methods to m- 
pervme and manage each dynamic syeteme. Effective control 
requires monitoring and sae rming rapidly c h an ging ritu a tio ni l 
dam from tbs environment in which the system operates, 
developing and evaluating planned ac ti o n s., and executing thorn 
ft ff* to achi eve desired goals. This requires the control 
system to a model of the operating environment end 

to interpret tensor date and planned actions in light of this 
— ■ « 
motif t ■ 

For operating environments too complex to model nang tradi- 
tional process control models, knowled g e b aaed qualitative 
andeb can provide an effective means of approaching the 
tasks of dynamic situation assessment and planning. 
However, existing Al-bamd techniques for data interpretation, 
situation ascammcnt, and punning cannot accommodate require- 
ments for operating in riim ticmnl situations where critical 
conditions and sumptions may be varying dy n a m ica ll y. Fur- 
ther, existing methods cannot easily accommodate the 
synchronization of interacting protea se s engaged in situation 
aeaemment, planning and execution control. 

Blackboard control systems [4. 2] provide a fim step towards 
solving these problems. Our analysis (Section 34) shows, 
however, that they leek the facilities needed to meet require- 
ments of real-time r espo nsi veness and hybrid, layered system 
architecture*. 

After analyzing the generic real-time heuristic control problem 
mote closely in Section 2, we present in Section 3 the 
Heuristic Control Virtual Machine, or HCVM. aa an ap- 
proach that considerably extends blackboard control architec- 
tures to solve problems amoriated with their shortcomings, 
the underlying idea of the HCVM is to cast a blackboard 
control architecture into an object-oriented framework [3] and 
then exploit the additional features provided by object-oriented 
programming to design solutions to the above problems. 

We show how flexible, even dynamically variable interaction 
modes between objects can be uaed to achieve real-time 
naponiiveneBS Using objects with standard interfaces aa 
Vrappcif around both heuristic and analytical procedures al- 
lows budding hybrid systems that integrate knowledge-baaed 


and conventional p roce m control components. Control reaming 
using explicit og time and temporal relatione about ac- 

tivities In kxA the controlled environment and the control 
system provides an approach to meet timrlmrm re q uirem e n ts. 

Finally, current experience suggests a reper t o ire of methods 
applicable to a broad domain of applications. Se rti o n 4 dis- 
cuses how the HCVM could provide a computational model 
for a more comprehensive software engineering environment 
for building p roce ss mana gem ent application!. 


2. Hie Process Management Problem 


Prooem contact* is with monitoring and ramming 

in dynamic system followed by planning and ex- 
ecuting actions to conted dynamic systems . Analytical tech- 
< nkpite employed by co nventi onal process control utilize math- 
ematical models of controlled s y s tems, sxprw ra rt in terms of 
differential equations of time-varying functions relating sensor 
data and controlled xarisbtes Conventional prooem control » 
limited to applications where each mo d eb and wffkieat sen- 
sor data are available. Dynamic systems with this property 
are often succertilly msnegii by human openton using 
heuristic e xper t a e intend of analytical reas oning, la fact, a 
gnat n«*«>yr of p raa m control sy stems operate in caviron- 
mentt where they tie complemented by human ope ra to r * 
Thb observation to view analytical and he uri s tic 

prooem control es complementary. We refer to process 
management as the comhinttina of both. 

Heuristic techniques play a dual role in procem management: 
Supervisory control applies heuristic judgment to make deci- 
uom about results elaborated by analytical or heuristic 
methods, sod direct control int erp r e ts s e n s or data to make 
and execute control de ri si ons . 

Applications impose two types of requirements on procem 
management sy s tem s. Functional requirements consist of 
functxxm and tasks to bt p e rf ormed. Performance 
injMrsmsnfr axmtnia tima sad ocher resvroe. -vailahle to 
carry out tasks. Jtm f - rf m r performance is a particular chal- 
lenge difficult to o verc om e by knowledge systems. 
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Several c Mra tch tka of intended application d om a in s in- 
ftn aioail and pafonnioci requirement* Wi can- 

genera* data abort many fewer sutnywune or component* 
men are teibead over space and tana, wch as 
attUim bants of data every few boon. Bath 

c^ttoBow ml dhcre ts data occur with often dynamically 
e ar y to g dam ratm and urg ency. Data an aanliable bacansa 
of p ^ v * ** 1 s e ra or and amunnkation failures and noise. 

Ffcacffcmai requirements concern monitoring the environment. 

HHBat, fr— and re-planning, plan execution 
contraC and u nr* " — ri *c and tfce system^ own ac- 

tivities. 

1* blontcorftaf. Tbs function inrhxhr 

• JfiwdnfiH dm ladfCMWit Receive incoming data; fli- 
ts* fas* iataip r at and abstract jpcomlng data far date 

critical event and alert generation to interrupt, 
urn i hie or j la impf corrent activity. 

• jfudwfui system performance: Inspect comet procam 
management a ct i v it ie s posnhiy in conjunctio n with 
nsolm of monitoring incoming data, to detect critical 
co n trol e v en t a and raise control alerts 


Real-time pe rfo r man ce cannot be achieved by merely optimiz- 
ing h ardwar e and mftwexe to operate fast. For timely 
mponms a proosm managemen t system baa to plan and 
m o ni tor tbs temporal perfor man ce of fee own a c tiviti e s 
together with those of its environment, btemptibility re- 
quires the system archi tect ur e to provide facilities for setting 
up tasks of variable and awn dynamically varying aiz* and 
for asanaging suspension and r—isprinn of tasks to transfer 
control. In te r nipti bflity contributes to a c hie v e timeliness by 

providing a mechanism. to po s tpon e tasks and move up others 
to mmplfts more ti me s tre wed tasks earlier. 


X Situation Assessment. The proces s m a nagem ent system 
djagaom the corrent situation of the environment; deter- 
mine how It evolved from previous history, and predict fu- 
ture development* determine goals to be achieved by proosm 
ma nagemen t activity. Situation assessment is dona dynami- 
cally, La. it extends, modifies and revises previous easement, 
and it will have to i n corporate new information when it be- 
comes available. 

X Artiom Planning. From results of Monitoring and Situa- 
tion Amemment, produce a plan of p roe ms management ac- 
tions to teach the goals. Plana will in c lu de conditional 
oedema with alternatives if conditions do not hold. 
Po-piannbtg modifies plans to adapt to dynamically changed 
situations in the environment (known when new data have 
been received) or the pr o c es s management system (when new 
assessment* critical events have been produced! 

4. Plan execution and Monitoring. Plans produced by Ac- 
tion Planning will be high-level descriptions yet to be com- 
piled into cmcutable instructions to effecton or the user in- 
terface. Monitoring plan execution determines whether actions 
are carried out as spsciflad and have anticipated effects. 

X Managing Own Activities. The system must allocate 
resources to the above tart* start their activitie* and moni tor 
their program To do so in a timely fashion, it must aarign 
prioritie* schedule activitie* and manage interrupts. 

The p e rf or ma nce requirements specific to real-time systems 
conc er n their r e sp onsiveness to changes in their environment: 

1. Timeliness. All scheduled activity must be completed 
sufficiently early to have its desired effect on the en- 
vironment. Any tart that co mp lete s snccsarfully. but 
too late to haw its intended impact violates the timelt- 


X In ters eptl bQlty. Intemiots susprori or terminate ac- 
tivities in favor of activating other* Interrupts require 
a model separating atomic, now-interruptible from com- 
post** interruptible tasks that is compatible with the 
timrltiw requirement. Composite tasks can be inter- 
rnptsd to give control to tasks of higher priority. In- 
twrapm are acoocapanied by strategies for orderly 
rmnmption of tasks that preserve the integrity of the 
overall task. 150 


3. HCVM: An Architecture for 
Real-Time Process Management 


XI HCVM Overview 

The Heuristic Control Virtual Machim (HCVM) k a 
knowledg e p r oofin g operating system designed as an environ- 
ment for en gineering real-time, knowledge-intensive p r ocess 
management application. The HCVM (see Fig. 3-1) co nsis ts 
of three collections of objects Tbs top level controller 
(TLCX the knowledge space and the data space. The 
knowledge space nnnrisre of knowledge handlers that em- 
body he uri sti c experti s e and analytical procedure* The ele- 
ments of the data space are data handlers carrying infor- 
mation obtained from outside communication or produced by 
knowledge handler* All activity of knowledge and data 
handlers is managed by the TLG 



heuristic cramot virtual uschinc 



Figure 3-1: HCVM Ov er v iew 

A Knowledge handler k activated by the TUC when (1) H 
has been triggered. (2) it has been scheduled for execution by 
the TLC and (3) is in a state where its < precondition> is 
true. Trigger conditions are pr o p o ri t i n e a about information 
stored in data handler* Executing a knowledge handler con- 
sists of carrying out its <body> that may result in changes in 
the data apace and communication with the environment ex- 
ternal to the HCVM. Information awed in data handler s 
changes when knowledge handlers produce such changes Ceg, 
after inferring a new diagnosis! or when new information 
from the external environment ie communicated to the HCVM 
and affseta a particular date handler (ag, after having 
received new maaor data! When a data handler hae received 
aome update information, it stay execute some internal code 
ta for cnmpl* check c o nsi s t* nry . perform sens s moo t hing 
and avenging operation on new and previous value* or 
qualitative abstractions from numerical value* 



The XL C repeatedly executes a cycle of farirokmg the 
knowledge, data and control manager module* The knowledge 
ttMlo tmif — curing knowledge handlers thst tow 
t o f n marked inatoMt by the control manager. Then the 
data manager carrtoa oat naalthtg changer in the data apace. 
The control nanyr maintaina a control plan that qperiftos 
which knowledge handkKa) axe to be executed next end in 
which order. To suite at a control plan, the antral manager 
may perform n native control p la nning abtnt tasks to be 
done in the external e n v ir o n ment and the HCVM. To do thto, 
the control manager may employ knowledge and data hand- 
ler* thus rastrictiag a aerka of control cycles to control 
lemoning 


3 a Shortcomings M BLackboard Architecterm 

Tbt HCVM m dmcribed above to ementiaUy a bis ek board 

control o t fck c n w M cast into an object-oriented program- 
ming framework. Knowledge handler* correapood to 
knowledge source* the data apace to a combined domain and 
control blackboard, end the TLC include* the facilities for 
blackboard control xmaoning. However, this doea not euffke 
to m eet the requirements described in the previous s ect i o n. 
We now explain the mechuntoms by which the HCVM con- 
siderably extends bkckbmrd control architactures in onrmretinn 
with the problem* they are meant to solve. 

Considering the requirements poeed in section 2, blackboard 

architectures have particular difficulttoe in handling the fol- 
lowing problems; 

I nter mpti hHity of tacks in any computational model to 
determined by the grain rise of non-interruptibie, atomic 
units. Knowledge mtricm are such units in bl a ckb o ar d sys- 
tem* Oboe a knowledge source to activated, it to executed 
until it terminate* A knowledge sources provides a facility 
to collect cloatlv triated e x pe rti se and procedures so that in- 
teractions b etw ee n knowledge sources axe restricted to leas 
closely related capabilities. This principle of organizing 

knowledge can be in conflict with timeliness and interrup- 

tibility requirements to structure activity into tasks of nif- 
fictontiy small grain size. 

Timeliness to difficult to achieve because blackboard control 
to excemively expensive when scheduling tasks does not re- 
quire ex t ens ive control reasoning, such as for Tasks that are 
known to be ermitnd in a sequence, one after another. 

Responsiveness to an incoming glut of data with dynami- 
cally varying information density and u rg ency to difficult to 
achieve with Mark board systems. Incoming data need to be 
analyzed fint before control planning can schedule activities 
to respond, implying at least two nr cution cycles of the 
blackboard system that even may be interrupted by high- 
priority alerts. 

Hybrid and layered system architectures when activities 
such as blackboard control, or those of individual knowledge 
sources are handled by conventional “control boxes” or black- 
board systems in themselves are difficult to build u black- 
board System* Interaction* including property inheritance, 
switching activation* from lower to higher level tasks and 
vice vers* and managing interrupts between different levels 
have to go through blackboard control cycles although this to 
often not an adequate mechanism to use. 


33 Synchron o us sad Asynchronous Interaction Modes ^ 


To help solve 
additional meet 


problem* the HCVM introduces several 
and feature* 


Baridae the bUckb om d mod* of t riggering. erhataHng and, 
then invoking knowledge handler* the HCVM provides rlt 
two additional modes of interaction between models* firs* 
Arscr jwxjtog* passfag act iva te s a module by wading it a 
ma m eg* as usual for object-oriented system* The secoad to 
rtmot* rombk * caff by which an obpet has another object ex- 
ecute a routine and then return a result to the caller. For 
the first two mode* activities of objects ranting than of 
ochexs axe not directly linked to the ac tim wl o by ct* In the 
blackboard mod* both activities are separated by ar bit rary 
other activities Bor the blackboard and the m en age pawing 
mod* invoking and invoked object may logically and physi- 
cally proceed concurrently, and their a c tivi tto s ate aot 
synchronised. For remote routine call, the caller sad calls axe 
synchronised by the caller being su spended until the the 
ctllee tetania control to readme the caller. 

The HCVM adda an additional feature to tfcwe staadanS 
mcchanJame of interaction between modules i ntera c tion raodee 
may vary dynamically. For example, a knowledge handler 
may be invoked by blackboard control triggering and activat- 
ing it, by some ocher handler sending it a me— ge 
(knowledge handlers tie no mod objects! or by some ocher 
handler invoking a routine (*g, to execute a sec of rale*) 
that the knowledge handler provide* 

Handler* in the knowledge space that are doc ra ppoaed to be 
invoked by blackboard control are realized is task handler* 
in the HCVM: A task handler to like a knowledge handler, 
but it lacks a < t rigger condition (see Fig. 3-lX 

Knowledge and task may have bodice built up from 

task handler* with control peering from one twk handler to 
another in a way that may vary dynamically (eg* by 
evaluating conditions *o decide which one to the next to 
execute! 

Although these facilities appear to be straightforward exten- 
sions of blackboard systems by i n corpo ra ting additional stan- 
dard module interaction mode* they open a dpafiaa t 
problem of d n ri gnmg coherent interrupt management 
For exampto, s knowledge handler may be interrupted after 
having finished some task handler. Resuming exec uti o n of this 
knowledge handler may occur after significant things* sug- 
gesting to not simply continue with the next prrrioariy 
scheduled task handler. A thorough dtocumicn of tsdnaqua 
far such capabilities goes beyond the scope of this pa p er. We 
note, however, that they provide flexible interrupt aches— to 
attend rapidly changing needs in an optimal way. 


34 Multiple BCVMe and Heterogeneous Handlers 

La rg e s eal s procras management applications — y toad to 
knowledge handler! of considerable sir* blackboard control 
that to extremely complicated, and I/O-communkarion that is 
difficult to handle within a single HCVM-obpct because of 
immense data rate* Each of these activities is beet pe rformed 
by a s ys tem providing all the facilities offered by an HCVM. 

The other problem of proc cra management applications is that 
knowle dge baa ed reasoning closely interacts with traditional 
pro c es s control equipment, such as PID-cantroties and ocher 
types of "control boxes". 
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with three types of 


The eb^ct-arimted nature of the HCVM allows quite eeaily 
to trees three aware by two approaches 

• Aa as HCVM to Jmt as objrot, knowledge awl trek 
handlers as well re a TLC or its components may fat 
Thrnimfvrs be fuH-fUdged HCVMa. As mentioned 
shoes, a problem beyond the s c o p e of this paper to how 
re re a q e intsraettone (reach as c ro p e ity inheritance awl 
In te r r up ts) bet w een Vdiaary" and HCVM- 
kmiwtodge/tuk handlers 

* A knowledge or tarit handler may encapsulate a phyu- 

csl p roc e s s control unit, when HCVM-interections with 
each a may consist in ag«ifn»if»8* »rif * g *f mff 

dels and aet p o in ts 


15 Oareressteatton with External Environ went 

Unfonaatn dynamic changes in an external environment with 
the p rop er t i e s described fat section 2 lead to two problems of 
dynamically changing nee ds for communication betw e en en- 
vironment and p roce re management system. 

* Varying infannatkm density of incoming data require 
proce re management to allocate varying ireonmre for 
buffering and preprocesring. Much of p re p ro t earing con- 
cents the monitor functiona of screening and compacting 
inpu t data, where reroening discards less relevant data 
re early re poauble. 

* Varying urgency cf incoming data n eeds re he deter- 
mriwd and anticipated by p ro c e re management for piart- 
ntag and re-pianning its treks and resource allocation, fat 
particular, communication capacity has to be dynami- 
cally adjusted to varying urgency so that high- 
bnsdwidth communication is only devoted to meamges 
of highest importances 

The HCVM provides four mechanisms of increasing com- 
plexity to support solving these problems; 

t l/fWwBiniiwiritw| j§ by ft 

Manager" execu t ed re part of the TLC-loop. For each 
TLC-cyck, the Cnmmuninnion Manager emptire all I/O- 
buffexs by distributing inenming data into the HCVM, 
and rending data fat output buffers away. This method 
to adequate when information densities and urgencies of 
communicated data vary approximately in the same way 
re the amount of time that is spent on procaring the 
ocher TLC-tasks. 

X IflI-cominiinication a done by a knowledge handler trig- 
gered when data with s peci fi ed properties (eg, sufficient 
volume and urgency) appear in I/O-buff era. This is ade- 
quate re long re there la no need to differentiate be- 
tween proc essi ng rates in different I/Obuffem 

X I/O<ommunication is done by several knowledge hand- 
tors f pe ctolfoo d to pro c ere data for particular I/O-buffen. 


M Control 1— aiming Support 


A reas fa uttou are carried out in tbs environment and toil 


effects are d b rerv sd in the prorare man ag eme nt syskre. 
Control nations are ex ec uted by procere arenagererent to in- 
fluence the co urt of domain actions. Sdf control seto u 
are those guiding the prorare management s yst em to pretona 
fee own tatov The trek, cf "managing its own a ctivities One 
section 2) recursively repeats monitoring, ritaatkm areerererec, 
planning and pt*n execntkxL 


The HCVM to weU-sotad to su p por t control reasoning. 
Knowledge handlew embodying explicit control knowledge we 
triggered when data handlers re cor d relevant conditions aad 
man for en v ir o nm ent and p roc ere management. A 
of such knowledge end data handler darees to be ixretanttored 
with qaatia g an HCVM oonsritutea a control framework that 
drives the control manager fat the TLC to select and arhadalf 
praam msnsgemrnt a c ti v iti e s Knowledge handtere could ci- 
ther thenredvea be of different grain-sire, or be oomposita of 
knowledge/task handtere focusing an particular dynamic ritaa- 
tiona. The control manager of the HCVM dynamically 
manages priorities and temporal de pen d en c i es among 
knowledge handlers scheduled for execution, providing a 
me c hani s m to re spond to dynamically changing n ee ds 


4. Mapping Architecture to 
Application Structures 


4.1 Multiple Agents Reasoning About Hierarchical 

Component Structures 

We have applied the HCVM to the development of a con- 
tinuous pro ce re management problem of mineral refinemras 
This domain has the following characteristics 

* The e nv i r onment consists of hundreds of componfstv 
with thousands of s e nsor s -gathering data about the 
states of components 

* Components are organized fat a functional hierarchy 
where interactions among functions coincide with control 
and data flow. 

* Each major component requires independent proem 
man agemen t within given requirements dynamically in- 
fluenced by other major components 



figure 4-1: Hierarchy of Procere Management Agents 

Fig. 4-1 shows a p roc ere management system structure whew 
a hi er a r ch y of knowledge handlers reflects the comp o nror 
hierarchy. Each knowledge handler is an HCVM-object per- 
forming process management for a particular com ponen t. 
Higher-level knowledge handtere combine sensor data wizh 
monitoring and situation imminent results from lower-level 
agents to fann more abstract, strategic amtawrun ta and plans 


4. I/O-communication is done by knowledge handlers ac- 
tivated both synchronously and asynchronously. 



Lower-level knowledge hiadtea C&g* KH-1, KH-2. - la Hg* 
4-1) axe *' T * rwtlJ la t fixed order which ie Imp l emen ted ae 
raqueutial elocution when trigger conditions are ignored. In- 
t er r u pt s, h ow ev er, may lead to t ermin a t e execution of aay 
g mfc tnfc* aul name execution of the sequence at any given 
poin t, ganaptku ia implemented ae triggering knowledge 
scheduling the fiat la the sequence. and diannhag 
the net from the control plan (ae subsequent dem e n ti of the 
sequence; they an execute d , anyway). 


4 L2 Towards Proceae kfaaaguaaeat En g ineer ing 


Knowledge/taak handlers ate activated upon the 
occ u rr en c e of changes in the data apace, or upon the 
penitence of p rop er ties about data over a specified 
p erio d of time. For situation nmnacut, 
knowledge/taak handlers determine situations and 
events constituting higher-level dcacriptioaa about 
co nditi o ns and changes fat the environment. Con- 
ditions and changes occur fat the contexts of tem- 
porally extended pattens of si tua tio n s and eveucs, or 
proc esses . Situation ameasment results fat recogaxzsng 
which prows are active, how they interact and to 
which states they have ptogrrmrri 


Environments 

Tha H CVM provides a ca apisfa wd model in a more com- 
pr ehand ve architecturs for real-time knowledge systems. as 
shown fat Hg. 4-2. fat this role; the HCVM is a software en- 
vironment to implement the profa&emuolving frameworks that 
support boilding to perform the generic fimetions 

of process described in Section X 

Example 

To descri be bow the HCVM supports such a system organiza- 
tion, we review how a real-time pr oc ea e management system 
implemented on an HCVM performs its tasks [6l 

Sensor data are received as separate streams of 
data packages in several input porta A Communica- 
tion Manager Case approach 1 in Section 2-5) turns 
data packages over to package-epedfic knowledge 
handlers for screening. Unless critical values are 
d e tect ed , packet handlers mod relevant and pasribly 
pro prorated data into the data apac e . 


Recognition of prows activate knowledgn'task 
handlers that plan adequate itsponsrs Ongoing plan- 
ning easy be interrupted and revised following alerts 
g enera ted when critical values era obse rv ed when 
data, or critical situations or 
events are inferred by situation smnumrnr Inter- 
rupts are frequently used to interleave situation ae- 
semment and planning. For example; planning may 
request more detailed information on the o ccuii ence 
of an anticipated critical event before committing to. 
on of several posable courses of action. 

The example indic a te how p roc era management functions axe 
accomplished by using gene ric capabilities, such as data 
abstraction, event and procea recognition, respo ns e planning, 
and control management. A software engineering environ- 
ment far p ro c es s management would provide the capabilities 
comprising the jsbhlcm-solving frameworks of Fig. 41 


Data handlers provide capabilities of a prow data 
base management s y stem: Data are steed in a way 
that supports retrieving historical information, such 
as “get the temperature distribution between the last 
sharp drop and aubnquent sharp inrwuss of 
premure*. Data h a n d ler s also perform data abstraction 
from numerical to qualitative values (quantization), 
smoothing and averaging operations, and locally 
decidable reliability checks. 



Figure 4-2 Organization of Real-time Knowledge Systems 
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5. Conclusions and Future Work 



W« bara ahowa bow tte HCVM ataadr bUckboud control 
udto c tt w to o mw a i many of their ihnrtminp Off 
mk h— si far indicated that the HCVM p twMw an ad»- 
fab o o — put a rt o n al —odd to sup port the variety of tab ia 
p ra— imrm,imnnt thus giving now evidenc e that it conld 
ha an a p prop ria te bade for future real-time prooa— manage- 
a — at environments. Tb become credibly this claim 

—oat be aoh—ntiated by actually building, applying and 
—hating axh ay—u— 

Aa with blackboard architectural the HCVM carriaa a dp 
aifkant c o mputati onal overhand for rim ply manag in g Ha own 
structure. Unlike blackboard awhitsmirea, howera; much of 
mm o— naan can at con—an ana even w— uj 
ey n rbrono ua In t era c ti on rood— betw e en handle— Empirical 
evidence needa to be gathered about the extent of each over- 
head. the imp ro v e m ent! achievable by le— expend ve inter- 
action modes, and the trade-offs with regard to I— ear degrees 
of seal-time i— ponaivi m— 

Another unexplored direction ia distributing knowledge 
(X 1] and data apaoea a ero — proc— am in dis tribat e d comput- 
ing environment*, A particularly interfering feature of the 
HCVM hi that it invotv— highly synchronized cnamonication 
via shared mem ory — well — loo— mr— age pan ring inter- 
action. Hardware architects x— a up po rting both t yp— of inter- 
action provi de appropriate auppoct for the— f— tuvan but there 
ia a wide s pec trum between tightly aynchronoua tod looariy 
aaynchroooua In tera c tions yet to be investigated. 
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ABSTAACT 

A hierarchical arcbitactura is described which supports space station telerobots in a 
variety of nodes. The systen is divided into three hierarchies; task deconposi tion, world 
nodal, and sensory processing. Goals at each level of the task decoaposition hierarchy are 
divided both spatially and teaporally into siapler connands for the next lower level. This 
decoaposition is repeated until, at the lowest level, the drive signals to the robot 
actuators are generated. To accoaplish its goals, task decoaposition nodules nust often use 
inforaation stored in the world nodel. The purpose of the sensory systen is to update the 
world nodel as rapidly as possible to keep the nodel in registration with the physical 
world. £he architecture of the entire control systen hierarchy and how 
it can be applied to space telerobot applications/^ ru d. € i ' . ‘ /' 


1. INTRODUCTION 

One of the najor directions on which the robot research connunity has concentrated its 
efforts is concerned with planning and controlling notion. Given a specific task, a notion 
plan aust be calculated which neets the task requirenents • Then, the plan nust be executed* 
there nust be sufficient control for the robot to adequately effect the desired notion. 

Trajectories are often planned as straight lines in Cartesian space (1). Whitney (2,3) 
developed the resolved notion rate control nethod for Cartesian straight line notions* 
Paul (4,5,6) used honogeneous coordinate transf ornations to describe a trajectory as a 
function of tiae, and Taylor (7) used coordinated joint control over saall segaents to keep 
the trajectory within a specified deviation of the desired straight line trajectory. 

While the research described above enploys a *kineaatic* approach to robot control, 
another direction of research takes the sanipulator "dynamics” into account in the 
description of robot notion. The dynaaic equations of notion are described either by the 
Lagrangian fornulation (8) or by the Newton-Buler equations (9) . Algorithas and coaputer 
architectures have been suggested which proaise real-time dynaaic robot control (10,11). 

Another aspect of notion control is concerned with the variables being controlled. The 
research described to this point was concerned priaarily with position control. The robot 
aoved froa an initial position to a goal position. While this is perhaps the aost coaaon 
node, there are aany applications for robots which suggest that other variables should be 
controlled. For exaaple, force control would be desired for asseably operations, taibert 
and Craig (12) suggest a method for hybrid position/force control of aanipulator s. 

These examples point to the sore general problea of sensory processing. For a great 
deal of robot notion research, sensory processing has been United to joint positions, 
velocities, and accelerations. However, other sensors are often required to accoaplish 
tasks. The control coaaunity has concentrated on the control aspects of the robot and as a 
result, little eaphasis has been placed on sophisticated sensory processing. 

Machine vision, an offshoot of image processing research, has recently been associated 
with advanced robot applications. One of the aost interesting directions in this research 
area is concerned with sensor controlled robots. Operating with the constraints iaposed by 
real-tine robot control, early methods used structured light and binary iaagea 
(13,14,15,16). These approaches, though developed at different institutions, shared aany 
concepts. One of the important subsequent research efforts vent toward the developnent of 
aodel-based inage processing. Bolles and Cain (17) used models of objects to guide the 
algorithas in a hypothesis/verification acheae known as the local feature focus method. Tha 
concept has recently been extended froa two diaensional (i.e. nearly flat) objects to three 
dimensional objects (18). Although the approaches described here have led to a better 
understanding of real-tiae vision processing, the systems lacked a sophisticated 
interconnection with the robot control system. 
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Tb« lutoatttd Manufacturing Research fwiuty (AM BP) * developed at the national Burma* 
of Standards* Is a hierarchically organised snail-batch netal machining shop (19). Zt 
separates sensory processing and robot control by a sophisticated world model. The world 
model has three complementary data representations. Lamia (20) describes the CAD- like 
section of the model. Shaeler, Kent* and Mansbach (21) describe the octree and table 

representations supported by the model. The model generates hypotheses for the features 
which are either aerified or refuted by empirical evidence. The sensory system* s task is to 
update the appropriate parts of the world model with new or revised data as rapidly ** 
possible. The control system accesses the world model as desired to obtain the current best 
guess concerning any aspect of the world. shneier* Lomia* and lent (22) describe the 
sensory system and its operation in greater detail. The AM IIP was the first deliberate 
attempt to tie together sensory processing* world modeling* and robot control in a generic 
fashion. The system developed for the AMRP is applicable to more than manufacturing* This 
paper describes its use in space telerobotics. 

2* A PUBCTIOMAL SYSTEM ARCHITECTURE 

• 

The fundamental paradigm is shown in Figure 1. The control system architecture Is a 
three legged hierarchy of computing modules* serviced by a communications system and a 
common memory. The task decomposition modules perform real-time planning and task 
monitoring functions* and decompose task goals both spatially and temporally. The sensory 
processing modules filter* correlate* detect* and integrate sensory information over both 
space and time in order to recognize and measure patterns* features* objects* events* and 
relationships in the external world. The world modeling modules answer queries* make 
predictions* and compute evaluation functions on the state space defined by the information 
stored in common memory. Common memory is a global database which contains the system's 
best estimate of the state of the external world. The world modeling modules keep the 
common memory database current and consistent. 

2.1. Task Decomposition - H nodules 

(Plan* Execute) 

The first leg of the hierarchy consists of task decomposition H modules which plan and 
execute the decomposition of high level goals into low level actions. Task decomposition 
involves both a temporal decomposition (into sequential actions along the time line) and a 
spatial decomposition (into concurrent actions by different subsystems). Bach R module at 
each level consists of a job assignment manager JA* a set of planners PI«(1)* and a set of 
executors EX(i). These decompose the input task into both spatially and temporally distinct 
subtasks as shown in rigure 2. This will be described in greater detail in section 4. 

2.2. World Modeling - M modules 

(Remember* Estimate* Predict* Evaluate) 

The second leg of the hierarchy consists of world modeling M modules which model (i.e. 
remember* estimate* predict) and evaluate the state of the world. The "world model" is the 
system's best estimate and evaluation of the history* current state* and possible future 
states of the world* including the states of the system being controlled. The "world model" 
includes both the M modules and a knowledge base stored in a common memory database where 
state variables, maps* lists of objects and events* and attributes of objects aad events are 
maintained. By this definition* the world model corresponds to what is widely knows 
throughout the artificial intelligence community as a "blackboard" (23). The world model 
performs the following functions: 

1. Maintain the common memory knowledge base by accepting information from the 
sensory system. 

2. Provide predictions of expected sensory input to the corresponding G modules* 

based on the state of the task and estimates of the external world. 

3. Answer "tfhat is?" questions asked by the executors in the corresponding level I 
modules. The task executor can request the values of any system vsrisble. 

4. Answer "What if?" questions asked by the planners in the corresponding level I 
modules. The M modules predict the results of hypothesised actions. 

2.3. Sensory Processing - 6 modules 

(Filter* Integrate* Detect, Measure) 

The third leg of the hierarchy consists of sensory processing G modules. These 
recognize patterns* detect events* and filter and integrate sensory information over space 
and time. The G modules at each level compare world model predictions with sensory 
observations and compute correlation and difference functions. These are integrated over 
time and space so as to fuse sensory information from sultiple sources over eztended time 
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intervals. Itwly detected or recognised «v«nts, objects, and relationships are entered by 
the N nodules into the world nod el coanon aeaory database, and objects or relationships 
perceived to no longer exist are renoved. The G nodules also contain functions which can 
conpute confidence factors and probabilities of recognised events, and statistical estinates 
of stochastic state variable values. 

2.4. Operator Interfaces 

(Control, Observe, Define Goals, Indicate Objects) 

The control architecture defined here has an operator interface at each level in the 
hierarchy. The operator interface provides a neans by which hunan operators, either in the 
space station or on the ground, can observe and supervise the telerobot. Sach level of the 
task deconposition hierarchy provides an Interface where the hunan operator can assume 
control. The task commands into any level can be derived either from the higher level ■ 
module, or from the operator interface. Osing a variety of input devices such as a 
joystick, mouse, trackball, light pen, keyboard, voice input, etc., a human operator can 
enter the control hierarchy at any level, at any time of his choosing, to monitor a process, 
to insert information, to interrupt automatic operation and take control of the task being 
performed, or to apply human Intelligence to sensory processing or world modeling functions. 

The sharing of command input between human and autonomous control need not be all or 
none. It is possible in many cases for the hunan and the automatic controllers to 
simultaneously share control of a telerobot system. For example a human might control the 
orientation of a camera while the robot automatically translates the same camera through 
space. 

2.4.1 Operator Control interface levels 

The operator can enter the hierarchy at any level. The operator control interface 
interprets teleoperation in the fullest senses a teleoperator is any device which is 
controlled by a human from a remote location. while the master-slave paradigm is certainly 
a type of teleoperation, it does not constitute the only form of man-machine interaction. 
At different levels of the hierarchy, the interface device for the hunan may change but the 
fundamental concept of teleoperation is still preserved. Table 1 illustrates the 
interaction an operator may have at each level. 

The operator control interface thus provides mechanisms for entering new instructions 
or programs into the various control modules. This can oe used on-line for real-time 
supervisory control, or in a background mode for altering autonomous telerobot plans before 
autonomous execution reaches that part of the plan. 

2.4.2 Operator monitoring interfaces 

The operator interfaces allow the human the option of simply monitoring any level. 
Windows into the common memory knowledge base permit viewing of maps of service bay layout, 
geometric descriptions and mechanical and electrical configurations of satellites, lists of 
recognised objects and events, object parameters, and state variables such as positions, 
velocities, forces, confidence levels, tolerances, traces of past history, plans for future 
actions, and current priorities and utility function values. These nay be displayed in 
graphical fora, for example using dials or bar graphs for scalar variables, shaded graphics 
for object geometry, and a variety of map displays for spatial occupancy. 

2.4.3 Sensory pr ocess i ng/wor Id modeling interfaces 

The operator interface may also permit interaction with the sensory processing and/or 
world modeling modules. For example, an operator using a video monitor with a graphics 
overlay and a light pen or joystick night provide hunan interpretative assistance to the 
vision/wor Id modeling system. The operator night interactively assist the nodel matching 
algorithns by indicating with a light pen which features in the inage (e.g. edges, corners) 
correspond to those in a stored nodel. Alternatively, an operator could use a joystick to 
line up a wireframe model with a TV inage, either in 2-D or 3-D. The operator night either 
move the wireframe model so as to line up with the inage, or move the camera position so as 
to line up the image with the nodel. Once the alignment was nearly correct, the operator 
could allow automatic matching algorithms to complete the match, and track future movements 
of the image. 

2.5. Common Memory 

2 . 5 . 1. Communications 

One of the primary functions of common memory is to facilitate communications between 
modules. Communications within the control hierarchy is supported by a common memory in 
which state variables are globally defined. 
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Bach nodule in the sensory processing, world modeling, and task decomposition 
hlararchlas raads inpats from, and writes outputs to, the coaaon aeaory. Thus each module 
needs only to know where in coaaon aenory its input variables are stored, and where in 
coaaon aeaory it should write its output variables. The data structures in the coaaon 
aeaory then define the Interfaces between the G, N, and ■ nodules. 

The operator interfaces also interact with the systea through coaaon aeaory. The 
operator displays slaply read the variables they need froa the locations in coaaon aeaory. 
If the operator wishes to take control of the systea, he siaply writes coaaaad variables to 
the appropriate locations la coaaon aeaory. The control aodules that read froa those 
locations need not know whether their input conaands derived froa a huaan operator, or froa 
the next higher level in the autonoaous control hierarchy. 

2.5.2 State Variables 


The state variables in coaaon aeaory are the systea* s best estiaate of the state of the 
world, including both the external environaent and the internal state of the I, K, and G 
modules. Oats in coaaon aeaory are available to all aodules at all levels of the control 

systea. 

The knowledge base in the coaaon aeaory consists of three eleaentss asps which 
describe tne spatial occupancy of the world, object-attribute linked lists, and state 


3. LSVSLS XV THE COST SOL HIEBABCIY 

The control systea architecture described here for the Flight Telerobot Systea is a six 
level hierarchy as shown in Figure 3. At each level in this hierarchy a fundamental 
transf or nation is perforned on the task. 


Level 1 tranaforas coordinates froa a convenient coordinate fraae into joint coordinates. 
This level also servos joint positions, velocities, and forces. 

Level 2 coaputes inertial dynaaics, and generates saooth trajectories is a 

convenient coordinate fraae. 


Level 3 decoaposes elenentary aove conaands (B-aoves) into strings of interaediate poses. 

B-aoves are typically defined in tarns of notion of the subsysten being controlled 
(i.e., transporter, manipulator, camera platform, etc.) through a space defined by 
a convenient coordinate system. K-nove conaands nay consist of syabolic names of 
elementary noveaents, or nay be expressed as keyfraae descriptions of desired 
relationships to be achieved between systea state variables. 1- moves are 

decoaposed into strings of interaediate poses which define notion pathways that 
have been checked for clearance with potential obstacles, asd which avoid 
kineaatic singularities. 

Level 4 decoaposes object task commands specified in terns of actions perforned on objects 
into sequences of l-aoves defined in terns of manipulator notions. Object tasks 
typically define actions to be perforned by a single aultiarned telerobot systea 
on one object at a tine. Tasks defined in terns of actions on objects are 
decoaposed into sequences of B-aoves defined in terns of manipulator or vehicle 
subsystem notions. This decomposition checks to assure that there exist arotion 
freeways clear of obstacles between keyfrane poses, asd schedules coordinated 
activity of telerobot subsystems, such as the transporter, dual ara aanipulators , 
nul tif ingered grippers, and centra eras. 

Level 5 decoaposes actions to be perforned on batches of parts 
into tasks perforned on individual objects. It 

schedules the actions of one or aore telerobot systems 
to coordinate with other machines and systems operating 
in the inaediate vicinity. For example. Level 5 
decoaposes service bay action schedules into sequences 
of object task conaands to various telerobot servicers, 
astronauts, and autoaatic berthing aechaaisas. Service 
bay actions are typically specified in teras of 

servicing operations to be performed by all the systems 
(mechanical and huaan) in a service bay os a whole 
satellite. This decoaposi t ion typically assigns 
servicing tasks to various telerobot systems, and 
schedules servicing tasks so as to maximise the 

effectiveness of the service bay resources. 
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Level I a*co«poM« th« MttHitt ttrTiclB 9 alssion plan Into atrfica bay action couinoi. 

Mission plans are typically specified in tarns of satallita servicing priorities, 
raqairaaants * constraints, and Mission tins liaa. Tba level I daconpositlon 
typically assigns satallitas to service bays, sats priorities for service bay 
activities, generates raquiranants for spara parts and tool kits, and schadulas 
tba activities of tba service bays so as to aasinisa tba af f activaaass of tba 
satallita servicing Mission. To a larga axtant tba level € Mission plans will ba 
ganaratad off lina on tba ground, aithar by bnaan Mission plannars, or by 
autonatic or saniaotonatie Mission planning natbods. 


4 . DETAILSD STEDCTDBE OP TH B NO DO LX S 

Tba ■ nodule at each level consists of three parts as shown in Pigura 4s a job 
assignnant Manager JA, one or nor a planners PL(s), and one or noi - executors IX(s). 

Tba job assignnant Manager JA is responsible for partitioning tba task conn and TC into 
s spatially or logically distinct jobs to ba parfornad by s physically distinct 
plannar/axacutor nechanisas. At tba upper levels tba job assignnant nodule nay also assign 
physical resources against task elenents. Tba output of tba job assignnant Manager is a sat 
of job conaands JC(s), s«l, 2, ...« ■ where M is tba nunbar of spatially, or logically, 
distinct jobs. 

Por each of these job conaands JC(s), there exists a planner PL(s) and a executor 
BX(s). Bach planner PL(s) is responsible for decoaposing its job conaand JC(s} into a 
taaporal sequence of planned aubtaaks PST(s,tt). Planning typically raquiraa evaluation of 
alternative hypothetical sequences of planned subtasks. The planner bypotbasisas soae 
action or series of actions, the world nodal predicts tba results of the action (s) and 
conputas none evaluation function BP(s,tt) on tba predicted resulting state of tba world. 
Tba hypothetical sequence of actions producing tba bast evaluation function BP(s,tt)nax is 
then selected as tba plan PST(s,tt) to ba executed by tba executor BX(s). 

P5T (s , tt ) - PL (a) JC {*) ,SF(s, tt)aax 

where tt is the tine sequence index for steps in the plan. tt nay also be defined as a 
running tenporal index in planning space, tt * 1, 2, ..., th where tb is the value of tba 
tt index at the planning borison. The planning borixon is defined as the period into the 
future over which a plan is prepared. Bach level of the hierarchy has a planning borison of 
one or two expected input task tine durations. 

Bach executor BX(s> is responsible for successfully executing the plan PST(s,tt) 
prepared by its respective planner PL(s). If all the subtasks in the plan PST(s,tt) are 
successfully executed, then the goal cf the original task will be achieved. The executor 
operates by selecting a subtask fron the current queue of planned subtasks and outputting a 
subconnand STX(s,t) to the appropriate subordinate B nodule at tine t The EX (a) nodule 
Monitors its feedback FB(s,t) input in order to servo its output .* .(s,t) to the desired 
subtask activity. 


STX (s , tvn) - BUS) PST (s, t) ,PB (s, t) 

where n - the nunber of state clock periods required to conpute the function XX(s) . n 

typically equals 1. The feedback PB(s,t) also carries tining and subgoal event isfornation 
for coordination of output between executors at the sane level. when the executor detects e 
subgoal event, it selects the next planned subtask fron the queue. 

Executor output STX(s,t) also contains requests for infornation fron the world aodel M 
nodule, and status reports to the next higher (i+1) level in the B nodule hierarchy. The 
feedback PB(s.t) contains status reports fron the fl nodule at the i-1 th level indicating 
progress on its current task. 

5. CONCLUSION 

This paper has described a hierarchically organised control systee and has shown how 
this generic systen can be applied to telerobotic applications in space by considering the 
requlrenents of e flight telerobotic servicer for the space station. 
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TAILS 1 


OFBKATOB ZHTXBACTIOS AT BACH LBVSL 


LSTBL 

At th« servo 
sbove level 1 
above level 2 

above level 3 

above level 4 
above level 5 

above level I 


TTPB OF 1STBSACTI0S 

replica master, Individual joint position, rate, or force controllers, 
joy stick to perform resolved notion force/rate control 

indicate safe notion pathways. ftobot coaputes dynamically efficient 
movements 

graphically or syabolically define key poses* nenus to choose elemental 
moves. 

specify tasks to be performed on objects. 

reassign telerobots to different service bays. Insert, modify, and 
monitor plans describing servicing task sequences. 

reconfigure servicing mission priorities. 
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FIGURE 1 s A llierarchial Control System Architecture for Intelligent Vehicles 



Task Decomposition 





ACTUATORS 


FIGURE 3: A six level Kierarchial Control Systcr. Proposed for 

Multiple Autonomous Vehicles 
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Abstnct — ' 

The General Piu>>se Automation Controller is a multi-processor 
architecture for automation programming. A methodology has been 
developed whose aim is to simplify the task of programming distrib- 
uted real-time systems for users in research or manufacturing. Pro- 
grams are built by configuring function blocks (low-level 
computations) into processes using data flow principles. These 
proce ss e s are activated through the verb mec h a n is m . Verbs are di- 
vided into two classes: those which support devices, such as robot 
joint servos, and those which perform actions on devices, such as 
motion control. This programming methodology was developed in 
order to achieve the following goals: 1 ) Specifications for real-time 
pr ograms which are to a high degree independent of hardware con- 
siderations such as processor, bus, and interconnect technology. 2) 

A ’’component” approach to software, so that software required to 
support new devices and technologies can be integrated by recon- \ 
figuring existing building blocks. 3) Resistance to error and ease of \ 
debugging. 4) A powerful command language interface. 

Introduction 

Recent system designs aimed at solving problems in automation 
control have made significant use of multi-processing [1-7). Typi- 
cally these systems incorporate a variable number of processors 
performing computations in parallel and exchanging data by means 
of some interconnect technology. These technologies are usually 
compatible either with/a shared memory model [61 of data exchange 
or with a message- pass mg model [5.7]; occasionally, systems may 
exhibit features of both models. If all processors execute the same 
program, the system is said to be SIMD (Single Instruct ton. Multiple 
Data), however the greatest flexibility is achieved with a MIMD 
( Multiple Instruction . Multiple Data) system. 

Multi- processor systems not only offer the prospect of increased 
computing power to meet the ever-increasing requirements of real- 
time control, they carry the potential for a high degree of 
configurability. One of the obstacles to rapid progress in robotics 
research and the deployment of programmable automation in scien- 
tific and manufacturing applications is the lack of configurability in- 
herent in roost currently available systems. The need for 
configurability arises from the need to integrate new devices, employ 
new strategies, or add processing power incrementally without 
making major changes to the system; for software this implies a need 
for "fast prototyping ", the ability to construct software that can 
rapkfy adjust to changing requirements [8.9 J. However, multi- 
processor systems cannot be used to tackle this problem unless an- 
other problem is simultaneously addressed: the lack of toob and 
methodologies to simplify the task of programming such systems. 

Whet a a P ro gra m mi ng M e thodolog y ? 

A true programming methodology is not simply a collection of tqob 
or techniques, rather it provides a set of concepts, usually formally 
or semt-formally defined, which serve as a basis for problem de- 
composition and program design. [10] This is a fairly definitive re- 


quirement which is not covered by vague terms indicative of a 
general approach, such as "top-down design" or "object-oriented 

A methodology has a number of advantages over a "toolbox", of 
which the following are representative: 

• The evolution or tools and techniques does not always promote 
ease of use, and of themselves toob do not help to guarantee 
good program design. [ 1 1 ] 

• A methodology provides a powerful language for describing the 
function of a program or specifying its design. 

• A methodology can provide a rnunework for reasoning about 
the correctness of a program. 

• Programs developed according to a methodology are often 
easier to maintain. 

This paper describes such a programming methodology developed in 
conjunction with a General-Purpose Automation Controller 
(GPAO project at IBM [8.9). The ultimate aim of this methodology 
b to simplify the task of programming real-time, distributed systems 
for manufacturing engineers, control specialists, and other system 
users whose primary area of expertise is not computer science. 

Currently, most distributed programs are written cither in a concur- 
rent language for which a distributed environment has been bag (e. 
g_, Ada [12.13]), a conventional sequential language with enhance- 
ments for distributed programming [4.7 J. or a special-purpose lan- 
guage [14]. Nooe of these approaches is particularly adept at 
dealing with highly reconfigurable systems. To change the behavior 
of a system one must make changes to the program’s source lan- 
guage. re-compile, re-link, re-download, etc. The usual approach is 
to provide primitives (15.16) (either as part of the la n guage syntax 
or as system calls) for communication, synchronization, mutual ex- 
clusion. and even control over the granularity of concurrency. Ap- 
plication programming in such a system involves not only getting the 
sequential algorithms right but also managing the interaction be- 
tween concurrent modules and using the primitives correctly. Many 
of the programs developed using these approaches are dependent for 
their performance, if not for their correctness, on a particular model 
of data exchange. Finally, there is no true methodology associated 
with these approaches, although useful toob such as debuggers, 
simulators, and syntax-directed editors are often provided. 

Basic Concepts 

The GPAC programming methodology depends upon a set of basic 
coocepts relating to both hardware and software. 

Umdmtne 

The fundamental hardware concept in GPAC b that of a real-time 
processor. Each real-time processor b a distinct entity which can 
perform one or more computational tasks. Each real-time processor 
has its own instruction stream, hence the GPAC model *s MIMD. 
A set of real-time processors sharing a common communications bus 
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or network is a real-time system (RTS). Programs are developed on 
a programming system (PS) and are then downloaded into the real- 
time processors. Associated with each real-time processor are one 
or more interrupt sources. An interrupt source, in the abstract, is 
simply a request to a real-tune processor to perform some work. 
Interrupt sources have priorities : work performed in connection with 
one interrupt source can be preempted by an interrupt source with 
a higher priority. Interrupt sources can be generated by other real- 
time processors or by devices connected to the real-time system It 
is important to reme m ber that the notions of interrupt source and 
priority are fundamentally abstract, and may be realized in the actual 
system in a number of ways. One last hardware concept is the 
physical device. A physical device is a gateway by which data can 
be passed to and from external hardware and is accessible by a par- 
ticular real-time processor. Sensors, actuators, pendant interfaces - 
to name but a few - are examples of physical devices. 

In GPAC. as in other systems ( I6|. implementation of the PS is de- 
coupled t om that of the RTS. so that the architecture of the PS can 
be quite different from that of the RTS. as it should be since the re- 
quiremerts are different. 

Function Blocks 

The fundamental software concept in GPAC is that of a function 
block [9). A function block is a basic computational unit assigned 
to one or more real-time processors; it communicates through input 
ports and output pvrts. In addition, a function block may communi- 
cate with physical devices, and may report conditions . or events which 
require exceptional action by the system. Finally, a function block 
may have some format parameters , which arc for its internal use only 
and normally not visible from outside. These five components 
comprise the interface to the function block. 

A function block can be viewed externally as a "black box" which 
takes inputs, performs some computation, and produces outputs 
(and in some cases, reports conditions). We denote the inputs, out- 
puts. devices, conditions, and parameters of a function block F by 
If Df. Cf . and F f respectively. 

The most basic form of function block is called an application sub- 
routine . In the current system, this is coded in the C language f 17] 
and corresponds to a C function. A set of macros is used to specify 
the interface; this hides any implementation details from the pro- 
grammer. 

An application subroutine is written in sequential code, and of itself 
contains no notion of concurrency. If certain coding conventions 
are followed, the application subroutine is also re-entrant, and mul- 
tiple instances of it may be active cither on the same real-time 
processor or on different real-time processors. A function block in- 
stance is obtained by binding the ports to specif»c data objects, sup- 
plying actual references for the physical devices, and values for the 
conditions and formal parameters. Other preconditions for the cre- 
ation of a function block instance are its assignment to a particular 
real-time processor and interrupt source, and determination of the 
means by which it is scheduled. 

Dt*m Flow Graphs 

More complex function blocks can be built up from simpler ones 
such as application subroutines. These complex function blocks are 
called data flow graphs. 

A data (low graph G consists of: 

• A set f of function blocks. 

• A set l 0 of input ports, a set 0 O of output ports, and a set of 
local cells. 


• A mapping p : / c U 0 6 U U — * where are the data 

nodes of G : 

*ttc- U tfUOf 

F'/c 

All the data nodes must be mapped to by p, i. e.. for any d e Ata^ 
there exists p such that d e p(p). 

A data flow graph is composed of communicating function blocks, 
but externally it is indistinguishable from an appficatioa subroutine. 
Data flow graphs are useful for expressing distributed execution of 
an algorithm. For example, a servo algorithm usually involves read- 
ing some value from a sensor, performing some computation, and 
writing another value to an actuator. In a distributed system, how- 
ever. there is no guarantee that the sensor and actuator will both be 
accessible from the same real-lime processor. Thus, in general, this 
algorithm cannot be executed by a single application subroutine, an 
instance of which is constrained to run on a single processor. 

Data flow graphs are also useful for pasting together application 
subroutines of general utility. Perhaps we wish to add some digital 
filtering to the input in our servo example. This can be done most 
conveniently by building a data flow graph, provided some digital 
filtering module has already been installed as part of the software 
component data base. 

Ports of Function blocks 

Each input and output port of a function block has a type and a 
mode. The type is simply a C type declaration. Ports can be bound 
to data objects only if those objects have a compatible type. The 
mode of a function block port describes the relationship between the 
modification, or updating of the object to which the port is bound, 
and the frequency with which the function block Is executed. 

There are three port modes: 

• synchronous - The object bound to the port is updated on every 
invocation. 

• ratio - The object bound to the port is updated at a specified 
sub- frequency of the frequency of invocation. 

• asynchronous - There is no fixed relationship between the up- 
dating of the object to which the port is bound and the fre- 
quency of invocation. 

Thus, if a function block has a synchronous input port, an instance 
of it can be scheduled for execution only when a new value of the 
object bound to the port is made available by another function block 
instance which writes the object through an output port. 

If a function block has no synchronous inputs, then it can be sched- 
uled by assignment of an execution interval or a trigger. The exe- 
cution interval specifies a frequency at which the function block 
must be executed: a trigger associates execution of a function Mock 
directly to occurrence of an interrupt source. 

Another rule enforced by the GPAC system is that values produced 
by a function block are not made available to other function blocks 
until the former has completed its current invocation. This bv« an 
atomic flavor to function blocks and helps to insure that the world 
will always be seen in a consistent state. 2 

Adva nt ages of Ike Fmmctiom Block Concept 

Specification of real-time computations using the function Mock 
concept has the following advantages: 

• Real-time requirements are separated from the executable 
code. This means that instances of the same code can be in- 
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voted wick dffering rnMne l eqw eme im, that nfateaeats 
can be rhingrfl dynamically. 

• Function bio cfcs have no knowledge, and conncquently no de- 
pendence. on the particular data exchange model. Although 
the current CP AC architecture is bared on shared memory, the 
system could be impfemented usin g message passing without 
affecting the design of ap p lications written for k 

• By sundan&ring the interface to function blocks sad hiding the 
implementation details, we hope to encourage programmers to 
design routines that sriD be easily reusable. One problem which 
has e merg e d historically in the "component" approach to soft- 
ware design is that, in the absence of a methodology, the suc- 
cess of the approach is dependent upon the good judgment and 
forest gru of programmers. 

• Consistency in data typing is automatically checked. Primitives 
for synchronization and mutual exclusion are mmecessary. 
hence their misuse is impossible. The result is a system that is 
more robust and easier to debug and maintain. 

Processes in GPAC 

Work done by a real-time system is normally divided into processes. 
In GPAC. these processes are merely sets of comm'micating func- 
tion block instances. Function block instances are installed before 
they can be executed; this consists of setting up the port bindings 
and attaching the function block instance to the proper processor 
and interrupt source. Consistent use of data objects bound to ports 
and of function Mock scheduling is checked during installation. 

A typical GPAC process will consist of three phases: 

• Installation of function block instances to be used. 

• Execution (either once, or repetitively) of one or more function 
block instances. 

• Waiting for termination of aQ the function block instances, or 
for a function block instance to report a condition that results 
in proc e ss termination. 

As an example, consider a process to implement coordinated motion 
of several robot joints. This process requires two function Mocks: 
one which compotes the coefficients of a trajectory (L e.. desired 
joint position as a function of time) based on the current positions, 
target positions, speed limits, etc., and another which generates 
intermediate points along the trajectory and outputs these com- 
manded positions to the joint servo modules. (The servo modules 
themselves are considered part of a different process, as we shall see 
shortly.) The First function block, called the trajectory planner, is 
executed once, the second, called the setpoint gen er ato r, is executed 
repetitively at some time interval (e. g.. 20 msec). The whole proc- 
ess terminates when the current (sensed) position of the joints be- 
comes close enough to the target position. 

Cammmd Lists 

We now describe the above process behavior using GPAC termi- 
nology 

1. Install the trajectory planner and the setpoint generator. 

2. Execute the trajectory planner once. 

3. Activate the setpoint generator for repetitive execution. 

4. Wait until the setpoint generator repons that the motion has 
completed (L e.. the sensed position is dose enough), or that 
some other condition (unexpected force sensed, time limit ex- 
ceeded, etc.) has occurred. 

This description of a process is called a command lot in GPAC. The 
elements of a command list are caDed function block commands and 
take function Mock instances as arguments. A process consists of a 
command fist and a set of termination conditions. 


Cmmumi Lkt Trmmkiam 

In the cane of robot motion, however, termination c o a tfi ti on a are 
often rimply signals that the system should proceed with the motion 
using a somewhat afferent plan. This k par ticu la rly true fat 
compliant motion and specialized co mb in atio ns of motions sack as 
centering grasp [8]. la these cases the termination of a process re- 
sults in a transition to another process. 

Ml 

We now have developed the necessary con c ep ts to define a in 
GPAC. A verb V contests of: 

• A set of function blocks. 

• A set pi of parameters. 

• A set <y of output values. 

n A subset test,- of py and a mapping x : hmf r -* 2**r where 
data i is the set of aO data nodes in V , L e.. the un io n of al in- 
puts. outputs, and physical devices in the function blocks of V: 

dbla, - U 'fU0 F UD r 

Kfr 

• A mapping + :pr - knt y — 2~*** where vab r is the set of all 
values accessible within K which includes all the ports as well 
as the conditions and parameters: 

rah, - U l r VO F VC F UP F 
r* fv 

n A mapping w : oy -•> nk r . 

n A set <v of command lists, of which one is distinguished as ini- 
tial. 

a A set t y of termination conditions. 

• A mapping r : mas, — c r where *aas r is the mwri ria ui of V, 
a subset of c, x /,-. 

• A mapping v : term, — 2*v . where term, is the ret of termi- 
nations of V : 

term,- - | f : 3r < c,- a {cj)4tnm % \ 

teli are the instantiation parameters at the verb, and x k the 
instantiation mapping. There elements describe the communication 
of data to and from the instantiated function blocks of the verh The 
verb parameters not in test, are called input parameters and the 
mapping + is the inpm mapping. These dements describe how initial 
values are set up to be accessed by the instantiated function Mocks 
of the verb. The mapping w is the output value mapping wh i ch de- 
fines a set of values that may be returned from the verb. 

Instantiation parameters are either data objects which can be bound 
to function block pom or references to physical devices. Input pa- 
rameters arc simply values which can be stored in the objects bound 
to pons or in conditions or parameters of a function block. As in the 
case with the mapping p for data flow graphs, every mem b er of 
data, must be a member of some ret m the range of x; that ri. every 
data node is mapped to by some instantiation parameter. 

If rir, j) - c 2 then if condition t occurs while the process described 
by r, is executing, that process is aborted and the process described 
by c, commences. If no transition is specified for a given termination 
condition and process, then receipt of that condition within the 
process causes termination of the entire verb. 

The mapping v de scrib es the termination actions of a verb. Each 
termination of a verb can return a subset of the output vafces de- 
fined for the verb. 
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The instantiation parameters of a verb provide bindings for the ports 
of the constituent function blocks of the verb. Thus, a p plic a tion of 
parameters to a verb yields function block instances for each of the 
constituent function blocks, and the coOection of these defines a verb 
instance. A verb instance in GPAC doady resembles the concept 
of a task in more traditional systems. Verb i nstances can be started, 
halted, suspended, and resumed. 

The distinction b e tw ee n verb and verb antance can easily be ap- 
preciated by analogy with an operating system utility r en t ing as a 
binary image on secondary storage. When the utility is invoked by 
a user, its inputs and outputs are bound to actual flies and devices 
and a task, or main memory image, is created. Furthermore, multiple 
instances of the same utility can sometimes be active in the system 
simultaneously. The same is true for multiple instances of a verb in 
GPAC. 

Verb instance commands are passed from the Programming System 
to the Real-Time System, where they are executed by the su p ervis ory 
software. In every RTS. there is one real-time processor which is 
distinguished as tile supervisor. The supervisory software is down- 
loaded (possibly along with applications code) into this processor, 
and it maintains communication with the PS while the RTS is run- 
ning its applications. While the PS sends verb instance commands 
to the RTS. the RTS sends notification of verb instance termination 
to the PS. 

Ail real-time processors, whether or not they are supervisors, con- 
tain a real-time kernel which is primarily responsible for handling 
interrupts, context switching, and dispatching of application sub- 
routines. The supervisor handles ad activity related to: 

• Installation, activation, and deactivation of function block in- 
stances. 

• Transitions between command lists. 

• Termination of verb instances. 

State Vector Variables and Logical Devices 

We have previously referred to the representation of data within 
GPAC without providing any details. Data which must be commu- 
nicated between function blocks is represented by the concept of a 
state vector variable (SW ). It is perhaps easiest to think of an SW 
as shared memory, but it need not be implemented that way. A state 
vector variable contains a buffer for the current value, and op- 
tionally a buffer for a set of preru ms values, in case a history needs 
to be maintained. Stale vector variables are bound to function Nock 
ports in order to effect communication between function block in- 
stances. Associated with each SW is a type, and it can only be 
bound to ports of the same type. 

Although a state vector variable is essentially an independent coo- 
cepc, the primary method of creating and using the SW in GPAC 
is through the morr powerful concepts of a logical device and a log- 
ical device type. A logical device type consists of: 

• An optional verb. 

• A specification for a set of SWs. 

• A specification for a set of parameters. 

A logical device can be considered an instance of a logical device 
type and consists of: 

• An optional verb instance. 

• A set of SWs. 

• A set of parameters. 

A robot joint is an excellent example of a logical device type. With 
each joint in the system we normaty want to maintain the following 
data: 


• Thecuncnt sensed position of the joint. 

• The current commanded position of the joint. 

• The goal, or target position. 

This (tea will be stored in SWs. since it wj| be updated by f auction 
Nock instances in real time. We may also asso ci ate certain parame- 
ters with a robot joint: 

• The maximum s pe ed at which the joint can be moved. 

• The maximum force or acceleration to which it may be sub- 
jected, etc. 

These values do not normally vary in real time and h en ce do not 
need to be stored in state vector variables. 

Logical Devices as Verb Parameters 

A verb may be applied to one or more logical devices: alternatively, 
we mig ht say that logical devices can appear as the objects of verbs. 
In this case, the state vector variables and parameters which com- 
prise the logical device will be entered into the parameter fat from 
which the verb instance will be constructed. Thus they can be bound 
to the ports and parameters of function blocks which wfl perform 
the computations associated with the verb. 

A robot joint is controlled by some real-time process (servo). This 
is implemented as a verb in GPAC. The definition of a logical device 
type for the robot joint contains specifications for the urvnmg verb, 
the SWs. and the parameters. Then, when an instance of a joint is 
created, the actual verb instance. SWs. and parameters are created 
for that device. 

Before a device may be used as the object of some action (e. g.. be- 
fore a joint may be mowed) it must be enabled. Enabling a device is 
equivalent to starting the verb instance which controls the device. 

Verb Composition 

Verbs may be composed, or combined to form new verbs. A com- 
position of verbs can be formally defined by taking unions of all the 
verb components; the verbs may then be sequenced by m inify ing an 
augmented transition mapping. In this way verbs with a highly spe- 
cialized semantics can be assembled out of simpler, mote general 
ones. (A verb with only one command fast is called simple. ) An ex- 
ample of this is "cenierutg grasp" found in (S| and J9J. 

Leeehmf Cumftwf 

The GPAC methodology supports programming for various levels 
of control. [9| 

• Closed-loop control is achieved through low-level real-time 
computations such as application subroutines and (tea flow 
graphs. 

u Concurrency control is concerned with specification of compu- 
lations executing in parallel and is achieved through configura- 
tion of command lists and simple verbs. 

u Sequent to* control deals with plans and strategies for executing 

complex (notions or tasks and is achieved primarily by verb 
composition. 

Programming for different levels of control involves different issues 
and areas of expertise: tins is reflected tn the methodology. 

User Interface 

AML/X 

The GPAC user interface is a program r unnin g on the Programming 
System Commands entered by a user are converted from a high- 
level form into actual messages to be seat to the Real-Turn System. 
Also, configuration of system hardware and software is done v» the 
user interface. 
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Because of the many abstractions and generic entities in the GPAC 
system, the high-level language AML/X was used to implement the 
user interface. AML/X ■» * iagay designed at IBM for auto- 
mation programming and other applications [18^. h has a number 
of interesting features, among which the three most relevant to 
GPAC are 

• Data abstraction capability. An abstract data type is called a 
classr. instances of classes consist of instance variables which can 
be manipulated using methods, a specialized type of subroutine 
caH The methods define the interface to the data type, except 
that some instance variables can be declared exposed, which 
makes them visible externally and hence part of the interface. 

• Operator overloading. Operators can be defined on dass in- 
stances. using a syntax similar to the method syntax. In par- 
ticular. the act of applying parameters to an object is regarded 
as an operator in AML/X; this allows GPAC verb invocation 
to have the same syntactic form as subroutine invocation. 

• Exception handling. AML/X has a rich set of constructs for 
raising and handling exceptional conditions (18). 

AML X can be easily interfaced to lower-level languages like C and 
FORTRAN In fact, the communication between the PS and RTS 
in GPAC is handled by a C subroutine package which is called from 
AML/X. 

Cottftgmnaiom ami Fveewtinm Pima 

The GPAC user interface consists of two parts: a configuration phase 
and an execution phase. 

In the configuration phase the hardware layout is described, in terms 
of real-tune processors available, interrupt sources, and physical de- 
vices attached to the processors. Next, previously compiled and 
linked object code modules are downloaded into the real-lime 
processors. ( For the most pan. these modules are simply libraries 
of application subroutines linked with a real-time kernel.) Then ge- 
neric objects such as function blocks, verbs, and logical device types 
are defined Finally, some instances of logical devices may be cre- 
ated and enabled. 

The execution phase consists primarily of invocation of verbs. Verb* 
are applied to devices to create verb instances, and commands con- 
cerning these verb i nuances are then passed to the real-time system 

Configuration and execution phases can be interleaved to a certam 
extent New function blocks, verbs, and logical devices may be de- 
fined at any lime, and existing verbs may be redefined This can be 
done while real-tune applications arc running (as long as no in- 
stances of the affected verbs are still executing) The goal is to 
provide a highly interactive programming environment in which 
real-time system behavior can be modified in a very flexible and 
dynamic way Thts meets the needs of both robotics researcher*, 
who wish to experiment with alternative control strategies and new 
sensor and actuator technologies, and manufacturing engineers, who 
are responsible for reconfiguring workccUs to meet changing work 
requirements. 

Modes of Verb Itnocmiom 

The full life cycle of a verb instance can be described as follows: 

• A verb ts invoked by applying it to a parameter kst. These pa- 
rameters are processed by the PS and the informal ion needed 
to create an instance of the verb is vent to the RTS. 

• The RTS allocates and fills ua the necessary data structures, and 
reports completion of tha procedure to the PS. 

• The PS vends a command to start the verb instance 

• The initial command lot of the verb o executed on the RTS. 
The verb instance continues to execute until some condition 


forces it to terminate or until a smtpemi or hal r message is re- 
ceived from the PS. 

• When the verb instance terminates, the RTS tends notification 
to the PS. Depending on the reaaon for the trrmsaarinn. an 
exception c ty be raised oo the PS. 

• Finally, the PS sends a command lo delete the verb instance. 
The RTS complies, cleaning up and de a llocatin g al data struc- 
tures. 

In the simp l es t mode of invocation, this entire scenario is carried out 
synchronously. The user simply applies a hat of arguments to a ob- 
ject of the verb dass. e. g: 

novel joint v goal , speed); 

The above operation will not complete until the correspond in g verb 
instance terminates in the RTS and is deleted. 

A more efficient, although more verbone. mod e of invocation is 
provided by introducing an asynchrooons phase: 

vi: BIND nove. as yncM joint tgoal. speed); 

/* other work can be done here */ 
vi .wai t ( ) ; 

a aync h is a verb method that results in creating and starting a verb 
instance, and returning a verb instance object without waiting for 
termination. At some later point in time, the verb instance can be 
waited for. and it is deleted after its termination. 

Finally, a verb instance can be created and then tmdnpiy invoked 
with new input parameters each time: 

vi: BIND wove. new instance ( joint ); 
vi .start (goal * speed) ; 

/* do something else V 
vi .wai t( ) ; 

vi . star t ( another goal. another speed); 

/* do something else V 
vi .wai t( ) ; 
vi .delete( ) ; 

Current Status and Future Work 

The GPAC methodology has been i mp le me nted in conjunction with 
several different hard* arc architectures. The relative ease with 
which applications can be ported back and forth between these ar- 
chitectures is ooe encouraging result of our work. We have also 
observed that real-time computations are easy lo program, require a 
minimal amount of debugging, and have predictable behavior once 
debugged. FtnaBy. the system can be easily reconfigured, new de- 
vices and processors can be added wuhout laborious chang e s and 
recompilations. 

We have used a certain amount of formalism to d cs crAe the con- 
cepts underlying the methodology rather than relying on configura- 
tion examples in AML/ X. We chose to do this became it is the 
formalism (not the current configuration syntax) winch is really 
critical to understanding the methodology, became we did not wish 
to require detailed knowledge of AML/X of the reader, and because 
the syntax itself is subject to considerable change and enhancement 
as we gam ex p erience with the system. 

In the future, mure sophisticated applications wilt be attempted using 
this methodology. We wil then be able to judge the efficacy of the 
methodology for practical problems m automation. Graphical tools 
will be added to simplify verb and data flow graph configuration. 
Knowledge-baaed enhancements and aaluraManguagr-hhe inter- 
faces are also contemplated. 
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1. Abstract 

The Telerobot Testbed is a hierarchically distributed processing system which is linked 
Logether through a standard, commercial Ethernet. Standard Ethernet systems are primarily 
designed to manage non-real-time information transfer. Therefore, collisions on the net 
(i.e., two or more sources attempting to send data at the same time) are managed by randomly 
ji eschedul ing one of the sources to retransmit at a later time interval. Although acceptable 
jfor transmitting noncritical data such as mail, this particular feature is unacceptable for 
{real-time hierarchical command and control systems such as the Telerobot. Data transfer and 
[scheduling simulations, such as token ring, offer solutions to collision management, but do 
not appropriately characterize real-time data transfer/interactions for robotic systems. 

/ Therefore, models like these do not provide a viable simulation environment for understanding 
real-time network loading. A real-time network loading model is being developed which allows 
processor-to-processor interactions to be simulated, collisions (and respective probabilities) 
lo be logged, collision-prone areas to be identified, and network control variable adjustments 
to be reentered as a means of examining and reducing collision-prone regimes that occur in the 
process of simulating a complete task sequence. The phase-one development results are 
presented%in thin p.iprr. Results include 1) the theoretical foundation for the network flow 
model, 2) an overview of the simulation design and constraints, and 3) the software design. 
Ultimately, the simulation will be used to examine potential loading problems as out-year demo 
performance improvements cause increased data traffic. The simulation will also provide a 
systematic means of managing resulting loading problems. 

^rr^TntToduct i on 

Distributed processing systems are becoming extremely common for passing mail between 
processors that are collocated in the same facility or separated by large geographic 
distances. Therefore, viable commercial systems have been developed that place processors on 
communication networks. For purposes of passing mail between processors, studies on Ethernet 
local network efficiencies have shown mean response frequencies on the order of 39.5 ms, with 
1 00\ of all traffic arriving by 200 ms (Ref. 1). Average utilization (on the order of 122 
bytes per packet) with 10 hosts (processors) on the net indicates mean arrival times on the 
order of 10 ms (Ref. 1). Under normal operating conditions, the above response times are 
excellent. If variable hosts require an intermediate protocol package to ensure message 
consistency, then an additional mean overhead of 7 to 10 ms is not unusual. Again, if mail 
passing is the primary occupation of the network, then 20-50 ms is perfectly acceptable. 

The Telerobot Testbed is presently employing an Ethernet (with DECnet protocol) system 
to facilitate interprocessor communication within the overall system computer hierarchy. 
Where normal mail passing finds response times on the order of 20-50 ms perfectly acceptable, 
robotic systems find delays in excess of 10 ms undesirable. The key reason why it is 

important to minimize signal delays is »otential control instabilities at the lowest level of 
the control hierarchy (i.e., the manipulator end-effector servo control level). In 

hierarchical command and control systems, commands must be passed from the system exec level 
through several intermediate control levels before they reach the servo control level. In 

the Telerobot, the operator acts as the system exec by confirming the automated task sequence 
with the AI planner (the next level). The planner forwards high-level task commands to the 
run-time controller (the third level in the hierarchy) where each task is broken down into a 
string of primitives containing important end-point state variables, trajectory via points, 
and force/torque information. The run-time controller then forwards requests and commands to 
the manipulator control and sensing/perception processors for control execution (the fourth 
level of the hierarchy). Additionally, because of mismatches between processor protocols, a 
network interface package is necessary to maintain protocol consistency over the net. Even 

though there is a discrete hierarchy for command passing, each processor will be 

simultaneously managing outgoing commands and incoming requests (i.e., requests for 
wor Id-state updates from higher level processors). In a static environment in which all 
fixtures are stationary, the effect of collisions on the net (two oi more hosts trying to 
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send data at the sa me time) will be to merely degrade the rate at which the manipulators aed 
end-effectors respond to incoming commands. However, in a dynaaic environaent in which 
objects or obstacles are moving (such as a rotating satellite), the degradation in speed 
could cause significant control p rob leas and potential damage to the manipulators and 
end-effectors. 

Techniques have been developed which ensure that network collisions are ainiaized. One 
of the most popular aethods is token ring. The token nng tecbnx qua basically assumes that" 
aessage packets arrive according to a randoa process (Ref. 2). A single control token 
circulates around the ring froa one host processor to the next. When a host observes that 
the token has been received, a data packet is queued for transaission. The token basically 
completes an array of “and" gate inputs and allows the packet to be transmitted. Upon 
completion, the token is passed to the next host, and so on, until it returns to enable 
another packet to be sent. Although this approach minimizes collisions on the net, it does 
not enable asynchronous network traffic (such as would be experienced by the telerobot) to 
propagate back and forth. For example, if an emergency stop signal was required in response 
to a calculated error, then control problems could arise if the processor that needed to send 
the signal had to wait a full cycle to receive the token. 

Therefore, in developing a simulation for the Telerobot command and control hierarchy, 
attention had to be paid to developing a more stochastic network event process. Stochastic 
petri-nets (Ref. 3) and stochastic activity networks (Ref. 4) provided the most fruitful 
basis for modeling and simulating message traffic on the telerobot distributed processing 
network. These models were useful because they basically model 1) the arrival of packets, 2) 
the queuing of packets, 3) the propagation of data, and 4) the detection of collisions. The 
Telerobot distributed processing hierarchy can be characterized as shown in figure 1. 



Figure 1 shows the operator control station (OCS) interfacing with the testbed exec 
(TBE) and the AI planner (A1P). During teleoperation the intermediate levels of the 
hierarchy are bypassed and the OCS/TELEOP interfaces directly with the manipulator control 
mechanization (ACM) and sensing/perception (SfcP) subsystems. The TBE also interfaces with 
all other subsystems. However, tor the FY 1987 and 1988 demos any robust control capability 
for the TBE will be suppressed; only system initialization and configuration will exist, 
which primarily requires a one-on-one interface with each subsystem in the hierarchy during 
start-up. At the AIP level of the hierarchy the planner interfaces with the OCS, TBE. and 
the run-time controller (RTC). Again, because of the hierarchical design, the TBE and OCS 
interfaces occur respectively during the system start-up and initialization phase, followed 
by the task sequence (menu) confirmation phase. The RTC interface occurs alternately as each 
task element in the sequence is presented to the operator for review/confirmation after the 
RTC has retrieved and forwarded the various world-state parameters to fill the menu. The RTC 
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interface is also initiated after the Menu has been appropriately coepleted and the AIP has 
forwarded the first set of execution c omma nds . The ETC then receives those commands and 
proceeds to pass specific control primitives down to the NCM and SAP subsystems. It should 
be noted that during task execution at the MCH and SAP level of the hierarchy (the servo 
control level) , any of the upper levels (i.e. f the A1P or KTC) can request status reports to 
Monitor task progression and update their respective world-state data bases. The reader 
should note the indicated interaction between the MCM and SAP subsystees. This interaction 
— dutiag exeaition <m~o. — mam U iM » iioyjy rotating satellite). 

During the execution of a task Macro, ro— nication with the upper levels of the hierarchy 
can be temporarily delayed to insure that cooMunication delays are MiniMised in order to 
prevent potential control instability associated with the dynanic environMent. For the 
rr 1917/1911 demos, the NCM and SAP subsy stows will interface with each other through either 
a parallel interface (in which case the net will be bypassed) or the planned DECnet network 
interface package (HIP). Figure 1 clearly shows that the area of concern, in terws of 
asynchronous requests, potential collisions, and subsequent delays in execution, is at the 
third level of the hierarchy (KTC). Table 1 su—t rises the various operational Modes, 
processor involvement, data transeission frequencies, and data rates as expected for the 
Telerobot Testbed. 


Table 1. Expected Testbed Processor PerforMance 


Processors 

Operational Mode Involved 


Freq. of Order of Mag. 

Data Trans. Data Rate (Bits/Sec) 


Start-up/Shut- OCS - TBE 

down TBE - AIP <10~ 3 Hx <10 2 bps 

TBE - KTC 
TBE - MCM/SAP 


Status/Initial- TBE - OCS 

ixation TBE - AIP , 

TBE - KTC lO^-KT 2 Hx 10 3 bps 

TBE - MCM/SAP 
AIP - KTC 
KTC - MCM/SAP 


* Planning 


OCS - AIP 
AIP - KTC 
RTC - MCM 
KTC - SAP 


.5 - 1 HX 


10 3 bps 


Execution 

AIP - RTC 
RTC - MCM/SAP 
TBE - AIP/RTC/ 

Mm /CLD 

RTC - 36 Hx 

10 3 -X0« 

bps 

- Autonomous 

MCM - 36 Hx 

V 

O 

«-» 

1 

m 

o 

H 

bps 



SAP - 5 Hx/50 Hx 

during vision 
servo ing 

X0«-X0 5 

bps 

- Teleop 

OCS - Teleop 
Teleop - MCM 
OCS - MCM 
OCS - SAP 

< S Hr 

10 3 bps 



Considering the above expected loading, the network activity is very low for the 
startr-up, status/initialixation, and planning operational nodes. This is because at any 
given time only two processors are actually talking to each other. This is one of the 
advantages of the hierarchical design. Furthermore, the comnunicat ion is at a fairly low 
rate and on the order of "quest ion- answer" type interactions. Since the bandwidth of the 
Ethernet is on the order of 10 Mbps, the network will only be utilised at a max of .001-. 01% 
capacity. The planning node similarly falls into the low utilisation category. As 
illustrated by Figure 1 and confirmed by the above table, the area of concern revolves around 
the autonomous control execution mode. Both the transmission frequency and data rates 
increase substantially. Even though the utilisation only increases to .1-1% capacity, the 
concern with this particular portion of the system in terms of modeling arises from projected 
substantial increases in the out-year utilisation. For modeling this critical area in the 
distributed processor configuration it appears that at s os w time in the task execution mode, 
three to five processors might be competing for access to the net. Also, the longer . the net 
is occupied by one processor, the greater the odds become that more than one processor will 
be competing for access to the net. The major overhead variables that surface from the above 







design and discussion (and confirmed by the literature (Kefs. 3 and 4)) are as follows: 

1. The nuaber of processors (hosts) attempting to communicate with each other at one 
tiae. 

2. The frequency at which the processors communicate. 

3* The size of the data blocks being communicated. 

4. The internal subsystem processing time per standard data block. 

5. The internal NIP protocol delay. 

6. The queue time for backlogged data packets before they get transmitted (from the HIP 
to the host). 

7. The retransmission delay resulting from a collision. 

8. The Ethernet transmission interval per data block. 

For this phase-one development act ivity , the simulation will only model network 
interactions involving the AIP, RTC, MCH, and SAP. The frequency at which the processors 
communicate or transmit data packets will be synchronized for task ezecution commands (i.e«, 
a data packet sent by processor n to processor n+1 must be acknowledged on receipt before 
processor n+1 can send a packet to processor n+2) and randomly selected for data update 
requests. The size of the data blocks 'being transmitted will be randomly selected within the 
respective bps ranges given in Table 1. The NIP internal overhead will be a constant (i.e., 
a mean value of 7 ms). The queue time delay will be 0 if the queue is empty, and increased 
accordingly as the queue increases based on actual experience with the NIP. The 
retransmission interval will be based on the Ethernet hardware specs and each retransmission 
time will be selected randomly within that interval. The internal processing time will be 
established by multiplying the specified average hardware internal overhead per data block 
times the randomly selected data block size. Similarly, the Ethernet overhead is determined 
by multiplying the inverse of the 10 Mbps times the randomly selected data block size. Since 
all processors are collocated in one facility, distance will not be a factor in the Ethernet 
overhead. 

3. Model Description 

The basic problem is to establish the time interval required to move an event (e) 
successfully from processor (n) to processor (n+1), which in turn must move the event to pro- 
cessors (n+2) and (n+3). If, at any time, the state processing and sojourn times for an event 
generated by a given processor are equal to (or overlap) the state processing and sojourn 
times for an event generated by an interacting processor, then a collision occurs and the 
respective events receive an additional retransmission time delay, are placed in a queue, and 
a new state is calculated/tested to determine if a collision occurs. The event must pass 
successfully through the four processors before it is discarded and a new event randomly 
selected. Before the event is discarded, the vital statistics of 1) total sojourn time, and 
2) number of collisions are collected and stored for calculating the overall system delay and 
collision probabilities. The events and interactions are structured as an input/output flow 
problem with each processor being linked by tagged events (e.g., an event that successfully 
leaves the AIP (processor n-l) as an output becomes an input to the RTC (processor n+1-2), and 
so on through the hierarchy). Mathematically, the overhead state time interval (S) for pro- 
cessor n and event 1 (e-1) can be stated as the sum of all internal processor state overheads 
<s Q ) : 


S -o 


EE 

e»l o-l 


n-l 

o 


( 1 ) 


where, the state overheads are the delays indicated in Section 2 above. 


By not suppressing the upper bound on the event summation, the total overhead time for 
processor (n) can be calculated using: 


u p 

EE 

e-1 o-l 


•o n - 1 


( 2 ) 


where (u) represents the total events cr commands and requests associated with a given task 
sequence. It then follows that the total delay (S tot) associated with events passing 
through the hierarchy (in this case only the AIP (n-l), RTC (n-2), MCH (n-3), and SAP (n-4) 
can be given as: 



( 3 ) 
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Conditionally, if an event arrival at a processor conflicts with another event inpot/output 
arrival at the saae processor, then a collision (and additional delay) is imposed. The 
following collision conditions hold for a given event arrival: 

either. 




I 

n-n+1 


( 4 ) 


or, given the Ethernet overhead (E) 






n-n+1 


( 5 ) 


The above later condition simply means that the net is occupied and processor n+1 cannot 
accept the command from processor n. The command from processor n would then be placed in 
processor n+l*s queue and assigned a delay interval commensurate with its place in the queue. 

Calculation of the collision probability for each processor, and the hierarchy as a 
whole, is done by merely employing standard statistical expressions using the logged 
collisions and total events processed. 


The last major constraint employed for the model was a definition of “real-time.** This 
was necessary since collisions are really only important if they indeed cause a subsequent 
degradation in system performance (i.e., speed). Therefore, after each event ripples through 
the hierarchy and the collision/time delay data are logged, the system delay is compared 
against the actual lag time of the system to determine if the lag time was exceeded. It is 
presently planned to use the manipulator control lag as the real-time baseline. For example, 
this means that if control lag for manipulator movement (under the planned operating speed) 
is 20 ms, then a collision aggregate delay of IS ms would not affect operating speed at all. 
However, as response and operating speeds increase with improvements in manipulator control 
design, acceptable aggregate delays might be driven down into the 2-3 ms range. 

The above model is still being examined from the standpoint of completeness and 
available data. As the software development proceeds and data bases are assembled, 
additional changes may be incorporated to further define the processor interaction 
environment. The next section briefly summarizes the software program being developed to 
implement the model. 


4. Description of Simulation Program 

In this section we will illustrate the construction of an event scheduling simulator for 
the network model discussed in the previous sections. In this simulator each event, created 
and scheduled with a time tag, represents the main activity of the network model. This time 
tdg decides the ordering of events in the event pool of the system. This ordering also 
determines the execution of the events and therefore describes the operation of the network 
mode 1 . During the operation, there is always one available event. 

As shown in Figure 2, the simulation program consists of six major components: 
initialization, event access, housekeeping, network activity, event handling, and report 
generation. The initialization module prompts the required system parameters such as number 
of desired RTC commands, initializes global variables, and sets the initial conditions cf the 
system. The event access module uses a time tag and event number for each event to select a 
most recent event for execution. The execution of an event means activation of the event, 
transmission of the communication packet, and processing of the command actions if any. The 
housekeeping module advances the system clock based on the time tag of the current event, 
generates data requests periodically, and collects the required statistics data from the 
network activity module. The network activity module evaluates the interactions (state 
overhead) of the available events in the Ethernet network. The event handling module 
performs the processing of the command actions to generate further activity in the system. 
It contains several handlers to deal with different types of events. The report generation 
module assembles the required system delay and collision probabilities once the simulation 
terminates. The system will terminate when the event access module cannot get an available 
event, which means the event pool is empty. 

Among these six modules, only the network activity module and event-handling module 
directly relate themselves to behavior of the network model. Therefore, they wilt be 
discussed in more detail. 
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Figure 2. The Main Flow Diagram of the Simulation Program 

_ The flowchart for the network activity module is shown in Figure 3. The main function 
of this module is to detect and record the collision of events. Collision occurs when two or 
more events are trying to occupy the network at the same time. Therefore, in order to detect 
the collision, the simulator computes the period of the current event residing on the network 
and with a given processor, and then searches through the event pool to check if any other 
events in the event pool will be arriving during this same period. If a collision occurs, 
all the involved events will be removed from the event pool and be appended to the network 
queues associated with the processor. Note that a random time delay will be added to each 
collided event for retransmission. If no collision occurs, the event becomes active and is 
ready to be processed (passed to the next processor) by the event handling module. The passed 
event is an output of the latter processor and an input to the next processor. 



Figure 3. The Flow Diagram of the Network Activity Module 


The flow diagram of the event-handling module is shown in Figure 4. There are four 
types of events in this simulator: command, acknowledge, request, and response. They will 
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be handled differently. Once a conund event is received, the subsystem will generate an 
acknowledge event back to the originating subsystem. Then the requested command is passed to 
the next processor. Concurrently, the AIP and the RTC processors use request events to 
acquire data from lower level of the hierarchy to update their own data bases periodically. 
The response event is the answer for the command and request events. Since the command event 
is executed sequentially, the completion of the current event also activates the next command 
event until all events are executed in the dummy task sequence. 
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EVENT 
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LZT 


RETURN 


SET COMPLETION TIME 
FOR THE CORRESPONDING 
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ACTIVATE THE 
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COMMAND EVENT TO 
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f RETURN 


Figure 4. The Flow Diagram of the Event-Handling Module 


5. Conclusions 

In this paper we discuss the simulation model for a real-time hierarchically distributed 
system using an Ethernet local area network. The probability and the impact of message 
collision on the network is the main interest. The conceptual design of the 'simulation 
program was completed. The implementation is under way. 

i 

Although the simulation is not operable at this time, a considerable amount of 
information concerning network design and operation has been obtained. This information has 
been used to design the Telerobot processor protocol and communication architectures in such 
a manner so as to minimize network interference. These design features include 1) a high 
bandwidth operating environment (i.e., 10 Mbps) to minimize the overhead on the Ethernet, 2) 
the overall hierarchical design which prioritizes and reduces the functions that lower levels 
in the hierarchy must perform, 3) inclusion of a special parallel interface at the servo 
control level of the hierarchy to facilitate off-net high data rate transfer during critical 
dynamic coordination tasks (e.g., satellite grappling), 4) adjustment of the Ethernet 
retransmission interval to correspond with processor priority relative to control execution 
(e.g., during error management, the RTC and MCM processors require greater access to the net 
than the AIP), and 5) the inclusion of a queue I/O in the network interface protocol package 
to accept incoming commands /requests even though the net is occupied. 

Using this simulator we can evaluate the system performance in terms of collisions 
during message transfer on the network, network utility, the data packet retransmission 
delays, and data request rate. For example, we can determine the response of the network to 
a utility level (i.e., transmission frequency), establish the lower and upper bound of the 
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data packet size, obtain an optimal data request rate, calculate new data flow control 
parameters to minimize collisions, and consequently resolve any system bottlenecks. 
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ABSTRACT . Computing in robotics has two important requirements: efficiency and flexibility. 

Algorithms for robot actions are implemented^ usually in procedural languages such as VAL aad J 

AL. But, since their excessive bindings ^create inflexible structures of computation, 

in t.M« rnirrri Logic Programming a*/ a more suitable language for robot programming dee 
to its non-de termini sm, declarative nature, and provision for metalevel programming. Logic 
Programming, however, results in inefficient computations. As a solution to this problem, ve- 
discuss a framework in which controls can be described to improve efficiency. JWe have divided 
controls into: (i) in-code and (ii) metalevel . and discussed them wijfch reference to 

selection of rules and dataflow. Vte hfw illustrated the merit of Logic Programming by 

modelling the motion of a robot from one point to another avoiding obstacles. 
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1. Introduction 



Computing in robotics requires both efficiency and flexibility. Large scale real time 
computation, both symbolic and numeric, is necessary for even apparently simple robot 
movements. Robot's actions must be as time-efficient as possible at the end-effeebor aad 
actuator levels[l]. At the same time, planning, backtracking, consulting knowledge about the 
world etc., requires flexibility. 


The world of the robot is dynamically changing, as in the case of telerobots, and the. 
change in the world must be continually noted. Accordingly, the planning of the robot; at the 
object and the objective levels has to be altered / readjusted. This is a strong case in 
favour of flexible computation for robotics at the object and the objective levels. 

In robotic computation, a third requirement is effective man machine communication [2]. 
This requires that the programming environment be as conducive to natural input and the 
language be as readable as possible. 

Traditionally, languages for robots are procedural, such as VAL[3], AL [*] and RAPT [5]. 
They provide for efficient computation, but are not flexible or easily modifiable. 

We propose that Logic Programming is more suitable for robot programming due to its nan- 
deterainism, declarative nature, and provision for aetalevel programming. Description of the 
world is most conveniently done in the declarative semantics of Logic Programming. Dynamic 

change in the world can be easily incorporated by the addition / deletion of logical 
assertions in the program. The fact that visual world can be best described declaratively has 
been exploited in [6] for declarative graphics. 

However, a major problem in using Logic for computation in robotics is its inefficiency. 
Typically, a Logic Programming language such as Prolog follows the depth-first strategy with 
chronological backtracking and a few control features such as cut to improve efficiency [7] 
However, recently, many features have been introduced in Prolog, to improve efficiency, such 
as intelligent backtracking^ , 9] , annotated variables [ 10] , and metalevel programming 1 1 , 12]. 

Of greatest relevance to robotic computation is the last feature vis., metalevel 
programming. Here, the specification of control strategy for the object level program (OLP) is 
expressed seperately, at a different level called metalevel. It permits one to intervene in 
the interpretation of the object level programs to define new strategies of control. It also 
allows one to specify one’s own interpreter. 

Since metalevel description is kept entirely seperated from logic level, the basic 
procedures and world description at the object level are left entirely untouched by efficiency 
considerations. Object level programs are still as declarative ani flexible as ever. An 
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additional element of flexibility is that the control strategy for any computation can be 
changed to suit the needs of the current goal and world. 

2^ in-code gad ttetfllflYfil Cgatrela 

For robot programming, we present control at two levels: (a) in-code, (b) metalevel; amd 
discuss them with respect to two types of controls * (1) selection control (ii) data flow 

control . 

In- code control refers to control expressed textual ly along with the OLP clauses and the 
syntax and the semantics are defined as part of the OUP language. At this level we provide: 

* efficient constructs which are procedural in nature. 

* liberal bindings of as much dataflow as possible, using the idea of locations. 
However, at the global level, we propose largely a declarative language. 

Metalevel control refers to control over selection of rules and is textually separated 
from OLP. At this level, control is expressed in the form of rules (different representations 
are possible) so that it is easy to modify when required. 

This strategy of splitting the controls has the following consequences on computation : 

1. Locally, the computation is expressed as efficiently as possible at the cost of 
flexibility. But, since it is only local, the resulting inflexibility may be acceptable as, in 
the worst case, it can always be replaced when modifications are required. 

2. Globally, the structure of computation is kept as flexible as possible, at the cost of 
efficiency i.e.,the programs need not be radically re-altered when the world or the Inpit 
changes for a robot.. Efficiency is improved by carefully choosing metalevel rules. Hence, 
readability and modifiability of programs are preserved, which is very important for robotic 
computation. 

The following are the control features we have Introduced to improve the efficiency: 
CONTROL FLOW: 

Incode: The scope of in-code controls is restricted to individual clauses and they are 

directed to specific predicates. 

(i) Forward Jump F(X) : Let F(X) be a predicate in the body of a clause C. The execution 
of this predicate results in the transfer of control to the predicate in the clause C, to 
which X is instantiated, skipping the predicates between F(X) and X in C. F(X) fails when X is 
uninstantiated or when backtracking. 

(ii) Backward Cut ! (XI : Let ! (X) be a predicate in the body of a clause C. !(X) succeds 
in the forward direction trivially. While backtracking, if !(X) is encountered, control is 
transferred to the predicate X occuring within the clause C. 

Metalevel: Several control features defined in Metalog[ll] actually fall under this: 

CHOOSECLAUSE, INHIBCLAUSE, I NB I BACK, FACTOR and BACKFAIL. In addition, we propose CB- 
parallelism as a control structure on the set of modules. This makes it possible to control 
parallelism by controlling the number of processes. An Important restriction we have placed 
here is that these controls are expressed with respect to clauses and not individual 
predicates. Thus the control here will tell which clause to use but not which predicate in a 
clause. Controls of this type affect the top-to-bottom selection of clauses for interpretation 
within a program. The scope of all the metalevel declarations is the entire program. 

(i) Q& Parallelism 

(PARCHOOSECLAUSE literal im.im, iTnl clausebody ) condition. 

The execution of this predicate results in the simultaneous selection of n clauses for 
interpretation in OR parallellism where the clause heads axe unifiable with literal i[j] 
denotes the J-th clause among the m clauses whose heads unify with literal . When n = 1, and 

i[l] = 1, PARCHOOSECLAUSE reduces to CHOOSECLAUSE of Metalog[ll]. 

DATAFL OW : 

I r code: In Prolog, data flow within a clause occurs through side-effects in the sense of [13]. 
We Introduce new predicates create and assign to let data flow both in the forward and 
backward ( on backtracking ) directions. This unlimited data flow within a clause enhances 
efficiency considerably . 


(i) create (LI ,L2 Ln) : Let C be the clause in which the predicate create occurs. 


•v Execution of this predicate in the forward direction results in the creation of locations Li, 
i s n. These locations can be used to store values using the assign predicate described 

^ below. Create fails during backtrack. r . : 

(ii) assign f L. express ion ) : Execution of this predicate results in storing the value 
of the expression in L, where L is a location. It fails when L is not created, and during 
backtrack. 

Hetalevel : At this level the rules define global data locations for the program. Data flow can 
occur across the clauses through parameters or global locations when it is convenient. 
Distributed computing for coordinated activities in a colony of robots can be achieved by 
appropriate metalevel descriptions of control and data flow. 

Sltful laggilsra 

GLOBAL L1.L2, . . . ,Ln. 

This declaration at metalevel defines n locations Ll,...,Ln that can be used as locations 
to store data and use them later in the program; they are visible over the entire program. 

SL ILLOSTRATIQH 

In this section, we address the problem of a robot moving in a world, avoiding 
obstacles. For simplicity, the world is divided into a number of squares. The obstacles are 
distributed in the world as shown in Figure 1. The start position of the robot is assumed to 
be the square (1,1), denoted by p(l,l). The target is the square p(6,4). We restrict the 
motions of the robot to horizontal and vertical directions only. 
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Figure 1: The robot world. 

The solution is provided at two levels: object level, and metalevel. At the object 

level, the mechanical actions and the simple heuristics the robot employs on encountering 
obstacles are described. At the metalevel, knowledge about the distribution of the obstacles 
in the world and the mechanisms for supplying information about local regions in the world ar ' 
described in order to guide the robot towards its target. The knowledge encoded in the objec 
level description is complete in the sense that the knowledge is sufficient, in principle, f o« 
the robot to reach any given target, by exhaustive search. Metalevel knowledge is incomplete 
in this sense, but it can greatly aid in reducing the exhaustive search, and thus lmprovir^ 
the efficiency. 

At the object level, we have the following heuristic to guide the robot: Let the current 
position of the robot be (Cx.Cy) and the target location (Tx,Ty), where Tx >= Cx and Ty >= Cy. 
If Tx = Cx and Ty = Cy, then the robot has reached its target, and therefore it halts; else, 
the robot moves to the square (Cx + 1, Cy) if Tx > Cx and (Cx ♦ 1, Cy) is obstacle-free, or to 
the square (Cx, Cy ♦ 1) if Ty > Cy and (Cx, Cy ♦ 1) is obstacle-free. Otherwise, it consults 
the expert in the current region. 

At the metalevel, the squares are grouped into regions. For example, p(l»l), p(l,2), 
p( 2, 1 ) and p(2,2) form Rl.(see Figure 1.) Each region has an expert who has the knowledge of 
the obstacles present in that domain. The expert has the ability to suggest a way out to 
each of its neighbouring regions, when consulted. For example, when consulted, R1 shows the 
way out to R2 via the square p(l,3). At the metalevel, we also have the following control 
heuristic to avoid infinite paths: In any region Ri , the robot ignores the advice of the expert 
of this region, if the advice leads the robot back to any of the previously travelled region 
Rj. In case of failure of a solution suggested by an expert, the robot backtracks to any 
alternative solutions that the expert may have suggested. If all the suggestions of the 
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current expert fell to provide an exit, the robot will take up any untried alternative 
suggestions of the earlier expert; and so on. 

The robot initially traces the following squares: p(l,l), p(l,2), p(l,3), p(2,3), 

p(3,3), p(4,3), p(4,4) . At this point, the robot is not able to wove further on its own, and 

so it consults R5. R5 suggests the war out to R2, R4 and R€. The war out to R2 is Ignored. If 
the robot accepts the war out to K4, via p(4,ij or p(3,2), the robot finds itself once again 
at a dead end. However, on backtracking, it tries the third solution suggested br R5, tints 
entering R6, and the target eventually 

/* Robot: Object level program */ 

/* Initialise stack for local expert and start */ 

toptrav( p(*X,*Y), p(*Xl,*Yl) ) create ( *list ), 

assign (*list, [] ), 

trav ( p(*X,*Y) , p(«l,*Yl) ). 


trav( p(*X,*Y) , P(*X,*Y) ) 2 - t 
trav( p(*X,*Y) , p(*Xl,*Yl) ) 

advance ( P ( *X , *Y ) , p ( *X1 , *Y1 ) , p( *X2 , *Y2 ) ) , 
trav( p(*X2,*Y2) , p(*X1,*Y1) ). 

advance ( p(*X,*Y), p(*Xl,*Yl), p(*X2,*Y2) ) *X1 > *X, 

*X2 is *X + 1, 

checkobst( *X2,*Y ), /* A built-in procedure that checks 

whether the square (X2,Y) has an obstacle »/ 
in_region( p(*X2,*Y), *R ) , /* Finds out the region to which (X2,Y) belongs */ 
test_append( *R, *list ), /* Checks whether *R has alreadr been traversed */ 
move( p(*X2,*Y) ) , /* A built-in procedure that enables the robot move br a square */ 
F( l ), move( p(*X,*Y) )# !( move ). /* The robot retraces its path 

in case of failure */ 

advance ( p(*X.*Y), p(*Xl,*Yl), p(*X2,*Y2) ) :- *Y1 > *Y, 

*Y2 is *Y + 1. 
checkobst( *X,*Y2 ), 
in-region ( p(*X,*Y2), *R ) , 
test_append( *R, *list ), 
move( p(*X,*Y2) ), 

F( i ), move( p(*X,*Y) ), !( move ). 

advance ( p(*X,*Y), p(*Xl,*Yl), p(*X2,*Y2) ) 

in_region( p(*X,*Y), *Rl), 

consult ( *R1 , p( *X, *Y ), p( *X2,*Y2 )). /* If met with a dead end, 

consult local expert */ 


/* To check whether R has been alreadr consulted */ 

test_append( *R, *list ) member ( *R, *list ), !. 

test_append( *R, *list ) append( *R, *list, *llstl ), 

assignC *list, *listl ). 

The metalevel description defines the local experts, and the position of obstacles. 
Associated with each expert is its knowledge regarding the position of squares which the robot 
should try to reach in order to avoid infinite paths. 

/* Local Experts: Metalevel program */ 

consult ( *R1 , p(*X, *Y) , p(*X2,*Y2) ) out( *R1. *Z ), 
member ( p(*X2,*Y2), *Z ), 
in_region( p( *X2. *Y2 ) . *R2 ) , 
not( member( *R2, *list ) ), 
advance(p(*X.*Y).p(*X2,*Y2),p(*X2.»Y2)). 

region* Rl, [ p(l,l). p(l,2), P(2.1), p(2,2) 3 )- /* Definition of region R1 »/ 
region ( R2. [ p(l,3), p(2,3), p(3,3) ] ). 
region( R4, [ p(3,l), p{3,2), p(4,l) ] ). 


out( R5, R2, [ p( 3 , 3) ] ). 

out( R5, R4 , i p( 3, 2) • p(4,1) ] ). 

out ( R5. R6. [ p( 5, 2) ] ). 
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/* Description of location of obstacles */ 
obst( 2.1 ). obst( 2.2 ). obst(. 3,4 ). 

obst( 5,1 ). obst( 5.3 ). obst( 5.4 ). 

L. fifettglaalsa 

In this paper, we have discussed the merits of Logic programing for robotic 
computations . Towards Inproving the efficiency of computation, while retaining the declarative 
nature of programing, we have split the control into two levels: in- code and nets level . We 

have proposed several constructs at both these levels, that improve efficiency. We have 
illustrated the elegance of metalevel logic programing and the usage of the predicates we 
have introduced, on a simple robot problem. 
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Abstract 

I InteUijeiU control of the Space. Station will retire the coordinated execution of computer program* toon * substantial 
I number of computing elements. It will be important to develop large subsets of these programs in the form of a single program 
which executes in a distributed fashion across a number of processors. Ii r thia p a per we de s cribe > translation stratep for 
i distri buted execution of Ada programs in which library packages and subprograms may be distributed. A preliminary version of 
the translator is operational. Simple date objects (no records or arrays as yet), subprograms, and atari j tasks may be referenced 
1 remotely. . { 

\ f v r , - Z « . 

1 INTRODUCTION 


Intelligent control of the Space Station will require the coordinated execution of computer programs across a substantial 
number of computing elements. It will be important to develop large subsets of these programs in the form of a single program 
which executes in a distributed fashion scores a number of proconors. The single program approach to programming closely 
coordinated mtions of multiple computers allows the advantages of language level software engineering developments, e.g., abstract 
data types, separate compilation of specifications and imple ment ations, and extensive compile time error c h ecking to be fully 
realized acrom machine boundaries. In this paper we describe one approach to a tra n s l a t i on system for distributed execution of 
Ada program. We consider loosely coupled homogeneous systems in which the program/processor binding is specified within 
the distributed program (static binding). 

There have been a number of proposals for distributed programming languages, |l), |2), |3], [4), (5), |6|, |7),it [«) to name but a 
few. Most of these language proposals hare emphasised models for communication and synchronization and/or a unified treatment 
of data abstrKtkms and multiprocessing that are amenable to correctness proving. With few exceptions, e.g. [7), however, they 
have considered neither the real-time aspects of the languages no f the full problem space (see [9)) involved in distributed execution. 
Only a few of these languages likely to see widespread adoption and use, though the important principals they lay down are 
likely he adopted in future language designs. 

Ada, on the other hand, will see wide sprea d use, and explicitly admits distributed execution. As yet, there have been only 
a few attempts at actually distributing its execution. Tedd, «t. at (10], describe an approach that is based upon clustering 
resources into tightly coupled nodes (shared bus) having digital c ommunic at i o n s among the nodes. They then limit the language 
definition for inter-node operations (e.g., no shared variables on cross node references); they are currently in the process of 
im plementing their approach. Cornhill has introduced the notion of a separate partiti o n in g l a ng uage jll] that can be used to 
describe how a program is to be partitioned after the program is written. [12] describes this language in greater detail. Again, 
neither of these approaches recognises the full problem space involved in the distributed execution of programs, [fij describes a 
number of difficulties which both approaches must face if they are to remain within the current Ada definition. 

[9| introduces three major d imens ions to the problem of distributed program execution: the memory access architecture, 
the binding time, and the degree of homogeneity. Tbs range of distributed execution systems that can be represented by these 
dimemioos is larger than any of the attributed language efforts to date, including Ada, can address. For example, one of the 
major dmign decisions must be in a distributed language is the units of the l a n guage that may be distributed. It is 
likely that one will want to the &tributabie unite a function of these dimensions. For ex a mp l e , one may wuat to allow 
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•hared variables for worm architectures aad disallow them for others. (IOJ does this by disallowing shared va ri a b le s across node 
boundaries. Yet the Ada Reference Manual is no* clear on this point. These are also questions of to what extent the specification 
of distribution should be part of the language (as o pp os ed to being stated in a separate configuration pha s e ); (Of di s c uss es these 
and several other issues, h as clear that the Ada language definition is real? not complete with respect to dirtributed execution. 

The work described in this paper as, thus, principally an experimental device to help identify the basic issues and point 
toward their solution. We restrict consideration to h om ogeneous loosely coupled computers with static binding specified within 
the program (one point in the problem specs indentified in (Of), and follow the suggestion in (Of and allow only ihraiy packages and 
subprograms to be distributed. We endeavor to avoid placing any further restrictions on the language, aad study the implications 
of effecting distributed execution within these constraints. This implies the need for remote access to data objects, subprograms 
and tasks. 

Rather than write a complete Ada translator, we adopt a pretranslator approach in which we translate a distributed Ada 
program ( - in our system a distributed Ada program s a normal Ada p ro gra m with SITE pragmas indicating where units are 
to be placed) into a set of normal Ada programs. We then use existing Ada compilers to translate each of the programs in the 
set. This approach has the dual goals of being a simpler experimental mechanism and ulXsing existing work where possible. It 
also has a few limitation*, which will be pointed out in the remainder of the paper. 

The next section presents an overview of the approach. Section 3 then discusses the critical problems and the details of the 
approach taken. Section 4 analyses the project performance of the method and section 5 summarises the current status of the 
work and discusses directions that must be explored in the future. 

3. OVERVIEW OF THE PROBLEM AND APPROACH 

The Problem 

We presume that the computers upon which a program is to be distributed are interconnected by a communication network, 
as shown in figure 1. Since we are allowing distribution of library packages and subprograms, our translation system must provide 
a means of accomplishing the following remote operations: 

• Access to procedures and functions declared in remote library units, 

• Reading and writing of data objects declared in remote library packages (and hence stored remotely), i-e., remotely shared 
data is allowed in our model, 

• Making [timed/conditional| entry calls on tasks declared in remote library packages, 

• Declaring/*] local mg (local) variables whose types are declared in remote library packages, 

• Elaborating tasks whose types are declared in remote library packages, 

• Managing task termination for tasks elaborated across machine boundaries. 

The approach 

The first issue that must be considered » the representation of the distribution. In our system, we write a single program 
and place a pragma called SITE before each library unit to specify the location on which that library unit is to reside. Far 
example, consider a mobile space robot system consisting of several mobile vehicles (each with a robot mounted on it) and sa 
overall system controller. If it were desired to have ooe vehicle controlled by computer number 2 and the overall control using it 
(as well as several other similar systems) placed on computer 1, a sample of the relevant code would look as follows: 

pragma SITE (2); 
package VEHICLE is 

procedure MOVE(..); 


end VEHICLE; 


pragma SITE(i); 
with VEHICLE; 
procedure CONTROL h 


begin 

VEHlCLEJdOVE(..); 


end CONTROL; 
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FifMif 2 ihrtratfi t he overall operation of oar system. A pre-translator traasUto oar aagie Ada propan with SITE 
pragmas rate a sst of independent Ada p rogr am whack iociadt fibrary asodaks of oar dcsiga to effect cooanw a kal aoa among 
proems m. TW ftraaslalioa system world i rpl a ct afl oBs (witkia CONTROL) to the procedara VEHICLEJdOVE(..) with tke 
appropriate rmota proc e d ur e cal. SkaBurly any reference s ia CONTROL to data objects or task entries d efined in package 
VEHICLE world be translated into appropriate remote references. E^di of tke individual program can thus be compiled by an 
Ada compiler. This ap pr oa ch simplifies the translation p ro cess con si derably since our pre-translator is ra n ch ten complex 
than a fall Ada compiler. 

Tke remote operations and in ter pro ces s communication are managed by a set of agents created for units that aa be remotely 
r eferen ce d and an underlying process to process mailbox system: During the pan through the pre-translator, all references to 
remote objects are replaced with appropriate reference s to agents. Each program unit (for example, suppose it is unit A) which 
might b~ -.rt 'enced from another, remote, unit (call it B) has three categories of agent associated with it. There is a local agent 
on the ‘ hot ting A, a remote agent on the site bolding B (and every other site referencing A), and a pointer agent placed on 
the same sii: as the local agent. A is essentially unchanged by the pre-translator, and both the local and remote agents can be 
generated from only the specification of A. AO references within B to code or data objects in A are translated into appropriate 
calls to the remote agent of A (which resides on the same site as B), which in turn uses the mess ag e system to pass the service 
request to the local agent of A. The local agent p erf or m s the necessary functions, returning any objects requested. The pointer 
agent serves to propagate object accesses involving access variables pointing to other sites, and is only required if such access 
variables are used. 

The relevant entities and flow of data for the example above are shown in Figure 3. CONTROL.T is the translation of 
CONTROL with the references to objects in VEHICLE replaced with calls into VEHICLE -REM-AGENT. 

5. TRANSLATION STRATEGY 

In order to solve the problems raised in the previous section the following issues must be resolved: 

• development of a general remote object accessing methodology, 

• translation of source code references to remote objects, 

• management of other remote service functions, e.g., creating objects, and 

• generation of the agents. 

The solution to these problems, while leading to reasonably efficient code, involves a rather complex set of multiple pass operations 
and the generation and use of a number of auxiliary files of intermediate information. Thus, a set of utilities ako are needed to 
allow the user to perform these operations in a straightforward manner. 

By far the most complex of these issues is the development of a general remote object accessing method. This is complicated 
by the need to address arbitrarily nested record and array components and the fact that component pointers may point to 
logically nested records or arrays on other processors. We thus concentrate our discussion on matters relating to object access. 
The solution to most of the other issues follows the resolution of this problem in a reasonably straightforward manner. 

The structure of the agents is critical to solving this problem, and is generally in three parts: 1) elements to access code 
objects, 2) elements to manage the address chain leading through qualified names of records and arrays, and 3) elements to 
manage other services. The interprocessor mail system and message structure is closely integrated with the agest structure. 

We begin this section with a discussion of the overall agent structure and its use for accessing code objects, aad then discuss 
related important issues of access via fully qualified names, the postal message structure, and the translation process. 

Agent Structure 

As mentioned in the previous section, three kinds of agents are generated whenever a library unit specification is encountered 
by the pretranslator: a local agent, a remote agent, and a pointer agent. Each agent is generated as a separate package, and 
assigned a unique name that is derived from the source package name. The agents can be generated simply from the package or 
subprogram specifications. 

The terms local and remote agent are used with respect to the processor holding the library unit which they represent. That 
is, the local agent resides on the same node as the unit it rep re sen ts, while the remote agent resides at each other node refer- 
encing the unit. Thus, a remote action of some kind begins with tke refe r encing unit making a call (after it is processed by the 
pre-translator) to the remote agent of the unit being accessed. For exampk, if the cell controller CONTROL makes a call to 
VEHlCLE.MOVE(..), the translated procedure CONTROL.T makes a call to the remote agent VEHICLE-REkLAGENT. We 
thus consider remote agents first. v 

Remote Apenis 

Remote agents are merely collections of procedures and functions that effect remote calk. In the case of subprogram and 
task entry calk, they present an interface to the calling package identical to that of the original source package, la our previous 
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example, the package VEHICLE REM. AGENT will emuia a procedure MOVE(..) that will receive the call i nt en d e d for 
packace VEHICLE. These procedures aad functions format u a p propr ia te msssage r e c or d (described below), and dispatch it te 
the appropriate rite via the postal servic e . When a return valve is r e c rivs d fron the local sgeui (on the other processor) via the 
postal service, this valve will be tetara sd to the cattng vait. from the p er sp e ct ive of the caBiag oaH, the facts that the actios 
is remote aad that there are (at least) two agents in b e twe en it aad the called unit are transparent, except for the longer time 
required. Thee, wo translation of sebprogram or (simple) task calls is rep ai red in the calling unit, vniess they use arguments 
raiding remotely. 

In the case of remote data object re ference s, a triads rent interface is not possible. Instead, a set of pro ce d u res to get aad 
put the valves of remote objects of various types is generated. Again, these procedures are generated from the specification cf 
the Ebrary package being referenced. In this case, the referencing unit (CONTROL in oar p re vi o us example) mast be translated 
to replace the object reference with a call to the appropriate get or pvt routine. Since the get aad put routines can be overloaded 
(with respect to the specific argument types used) the translation is straightforward. The specific arguments vaed aad the detailed 
actions of the get and put routines are closely intertwined with the siinsgfmrnt of folly qualified names, aad will be discussed 
later. 

Two other points are worth mentioning. Since the structure of the remote agent was chosen to rntn im i ae the impact on 
the referencing unit, the translation required by the pvetranslator is a minimum. Also, since sending and receiving messages 
from another processor is time consuming (relative to normal instruction processing times), the point after the tra n s mi ssion of a 
message is treated as a sychrouixation point so that other taoks may obtain the services of the epu while the reply to the message 
is in progress. 


Local AftnU 

The local agents are the most complicated of the three agent types. Their task is to service requests from remote sites needing 
to access data objects, subpro g ra ms , or task entries. A local agent consists of N+2 tasks where N is the total number of functions, 
procedures, and task entries, contained in the source specification of the unit the agent is helping to represent. One of these tasks 
is associated with each of the aforementioned subprograms and task entries. 

One of the remaining two tasks is designated as the local agent main task. This task consists of a single loop that requests 
message records from the postal service (via a task entry call), interprets the request, and dispatches the request to the appro- 
priate handler (task or procedure) within the local agent. Messages requesting access to data objects, are serviced immediately 
within the main task by calling a GETPUT procedure (described below) and an immediate reply is sent. 


with PACKAGE-NAME; package being represented 

task body AGENT-MAIN is 
M: MESSAGE-TYPE; 
begin 
loop 

POSTAL.MAIL-BOX.GET(M); - - get message 

case M.OBJ-ENUM Is - - Branch according to object name. 

- - Object references 

when NAME-1 => GETPUT(M, PACKAGE-NAME.NAME-1); 

when NAM ELK => GETPUT(M, PACKAGE-NAMELNAMEJC); 

- - Subprogram calls and task entries 

when NAMEJC1 => MANAGER.DEPOSIT-NAMEJCl(M); 

when NAMEJf => MANAGERJ)EPOSIT-NAME-N(M); 

end case; 

SEND-RETURN(M); 

end loop; 

end AGENT-MAIN; 

The above abstraction is only for a single distributed package. Actually, the message type would be embedded in a yet mom 
general record having a varient part for each distributed package, and the actual code would be slightly more involved. 

It is imperative that the main task not be blocked for it provides concurrent access to all objects and types in the specification 
of the unit it represents, and if blocking occurred here, other, parallel, requests could be delayed. In particular, the agent mut 
not be blocked by a unit it calls on behalf of a remote cBent, as could occur if the agent directly called the unit (the subprogram 
called might, for instance, become blocked on an I/O wait). That is why a task is associated with each subprogram and tii 
entry. The main task places the message r eceive d in a buffer, by calling an entry in a buffer manager task (the last of the tasks 
in the local agent). A flag counter corresponding to the requested eaC V *lso incremented at this time. 
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Th e buffer amyr task has N ktftioael otriu, wbost acceptance! in conditio— d ot % positive nhu of each of the 
co rresponding N coasters (indicating tkit there is & nsuijt to bo retrieved).. Three ore N cell manager tasks (the N tasks 
cor re sp ondin g to subprograms sad task entrire), whoso sole purpose h to retrieve a message record from the co rre sp onding 
conditional task entry in the buffer manager task, execute the appropriate cal to the source package body, and then return a 
reply to the remote site. 

v The following code abstraction illustrates the manager task and the tasks c o rresp o nd in g to the subprograms and task entries 
that may be tiled, 

task MANAGER Is 

entry DEPOSIT_El(MESG : In MESSAGE); 
entry DEPOSIT J51(MESG : In MESSAGE); 

entry EXTRACT-El(ME$G : out MESSAGE); 
entry EXTRACT J5 1 (MESG : oat MESSAGE); 


end; 

task body MANAGER Is 

E.FLAG: army (1.. MAX-ENTRIES) of INTEGER; 
begin 

loop 

select 

accept DEPOSIT-ElfMESG : in MESSAGE) do 
- - deposit the message for el 
£LFLAG(1) := E-FLAG(l) + I; 

end; 


when E-FLAG(I) > 0 => 

accept EXTRACTS 1 (MESG : out MESSAGE) do 
- - extract a message from the buffer and return it 
E-FLAG(l) := E-FLAG(l) - 1 

end; 


end select; 
end loop; 
end MANAGER; 


The suffix El indi c a t es the Ith entry point, and the suffix SI indicates the Ith subprogram. The structure of the entry task for 
entry El is as follows: 


task DCLEl 
begin 
loop 

MANACER.EXTRACTJE1( ); 
El( ); 

- - send back message; 
end loop; 
end DO-El; 


The imresy are provided by a mailbqat system that de li v e r s messages to the correct local agent. The message interpretation 
and task calls by the agent essentially achieves a routine to routine communication between routines in the remote and local 
agents in a way that prevents delays in the response to ooe request from locking out other parallel requests. 

Pointer tftnU 

Oar allowed model of distribution permits access variables to point to objects on machines other than the one holding the 
access variable. Since access van shies, as defined within a local Ada program, dearly cannot contain both the machine identity 
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a ad an tddrca, whenever an accem type definition is encountered in the source package, it is replaced by a record structure 
containing two fields: a site number, and the original access type. This new record type is then used in place of the access type. 
The site number is always checked against the current site number, to determine whether the object being pointed to is on the 
local site, or on a remote site. 

Because access variables can be passed from one machine to another, it is possible for a processor to bold an access variable 
pointing to an object on a site which it does not directly reference and for which it does not therefore have an agent. We therefore 
include pointer agents to allow access to objects on remote sites. The structure of pointer agents is similar to that of local agents, 
except that provision for subprogram calls need not be made. 

Remote Data Object Access 

Three characteristics of Ada data objects cause difficulty in developing a general mechanism for handling references to remote 
objects: I) the objects may be composite objects, 2) they may have concatenated names, and 3) parts of a fully concat en a t ed 
name may be access variables pointing to objects on other machines. 

The first issue manifests itself when one must copy a composite object (as opposed to a component of the object) from one 
site to another. For example, suppose that site 2 uses a record A on the right hand of an assignment statement and that A b 
located on site 1. Eventually, the agent and message system must convert A to a bit string for transmission. It would usually 
be desireable that the part of the system that performs the conversion not be aware of the structure of the object (from object 
oriented design principles). However, if the object contains a memory address as part of its structure, the result received could 
be meaningless. For example, suppose the record A contains a variable length array, as shown below. 

subtype S is INTEGER range 1..MAX; 
type IA Is array (INTEGER range <>) of INTEGER; 
type R(L: S := 1) is 
record 

B: IA(l..L); 

C: INTEGER := 0; 
end record; 

A: R; 

One decision for the memory allocation fox record might be to allocate the storage for the array from a heap and place only 
a pointer to the array (or possibly its dope vector) in the record. The need to perform whole object (record) assignments in 
Ada might discourage such a memory allocation scheme, but nevertheless, it is certainly a possibility. A bit by bit copy of the 
block of data corresponding to the record A, would then copy this address, which would have no usefulness when received by 
the requesting unit; in particular, the bit by bit copy of the record block would not result in the array values bring transmitted. 
To avoid this problem, the routine that does the final message transmission must, indeed, contrary to the above assumption, 
have knowledge of the record structure so that the array values themselves may be transmitted, and not just the address of the 
array. Since we are describing a pre-translator approach that uses existing Ada compilers, this knowledge is dependent upon the 
implementation of the underlying compilers used. 

To see the second issue, suppose that site 2 contains a statement like X := A.C . How does one construct aa address for A.C? 
Or describe, in a general way, to the agents what element is to be returned? The syntax *A.C” exists only on site 2, and the 
only information available there from the specification of the package containing A is the logical record structure of A, not its 
physical structure. Again, implementation dependent knowledge of the rules used for construction of the physical structure of 
records b necessary. 

If one were to now add a fourth component, D, to the type R above, that is an access type, and if the value of A.D were 
to point to another record stored on site 3, the third issue arises. The method used to calculate the address of the item to be 
retrieved must not only contain implementation dependent knowledge, but it must be dbtributed as well. 

Strategies for Remote Object Access 

We are studying two methods of obtaining composite (as well as scalar) objects: 1) using knowledge of the rules for storage 
allocation and physcial record and array construction, develop the dbtributed algorithms for calculating the address of the target 
object and then implement these, possibly in assembly or some other lower level language, and 2) use minimal implementation 
dependent knowledge and the logical structure of records and arrays to utilise standard Ada mechanisms to perform the object 
transfers. We expect the former to lead to more compact (in terms of code sixe) solutions, but to require a more detailed knowledge 
of the internal workings of the underlying compilers, while the latter will require less knowledge of the internal mechanisms used 
by the compilers at the expense of a larger amount of code (automatically generated, however) in the agents. Since the latter b 
also more in keeping with the philosophy of using exbting compilers where possible with minimal knowledge of their internals, 
and since developing thb approach will aid in developing the algorithms for the first approach, we have fol l owe d thb one first, 
and it b thb one that will be described below. In subsequent work, we will explore the direct calculation of object addresses. 
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Access to remote objects is based upon the following things: 

• An enumerated type, TJ5NUM, whose values ace the names of every type and field declared in the package for which an 
agent is being generated, and those in packages included via a with. 

• An enumerated type, N-ENUM, whose values are the names of every data object declared in the package for which an agent 
is being generated, and those in packages included via a with. 

• A collection of GET PUT procedures, one for each record or array type defined, whose functions are to either handle the 
request for an object reference if the request is for an object of the type the GETPUT handles, or to call another GETPUT 
if the object requested is, or is derived from, one of the fields of the type. 

• A variant message structure containing appropriate fields indicating the type of data required, the fields withm records to 
be used, and an actual data object of the type being referenced. 

From the perspective of the local agent, a remote direct (not via access variables) data object access begins with the local 
agent main task receving a message from the postal system. One of the fields in this record contains a value of type N.ENUM 
that indicates the outermost name in the fully qualified name of the object being referenced. The local agent main task then 
performs a case statement on this value. There is thus a case for each object name. Each case calls a GETPUT procedure and 
passes it the message, the object named, and a count of the number of name components to the fully concatenated name sought 
(including array arguments). 

If the object passed is a scalar object, the count will be xero and the request can be satisfied dirccly by * he GETPUT ~ 
procedure by simply copying a value between the appropriate field in the message record and the object passed to it. Another 
field in the message record contains the type of the object to be returned. 

If the COUNT is not xero, then either an array element » being sought, or a fully concatenated name has not yet been fully 
expanded. In the former case, the indices for the array element (or slice) are contained in other fields of the message record and 
the GETPUT can select the appropriate element(s) of the array. These either directly satisfy the request or are used to recurse 
as described next. 

If the GETPUT is handling a record type, there will be another field in the message record corresponding to this type of 
record which will contain a value of type T-ENUM (containing the field name to be selected). The GETPUT contains a case 
statement conditioned on this field indicator. There is th*.s a case corresponding to each field possible in the record. The action 
of each branch of the case is similar. Another GETPUT is called, passing to it the message record and object pointed to by a 
concatenation of the object name passed in and the corresponding field name. 

Below is an abstraction of a typical GETPUT routine for a record type. The forms for other types are similar, but tend to 
be even a bit simpler. 

procedure CETPUT(M: in out MESSAGE; OBJ: in out T; COUNT: NATURAL) is 
begin 

COUNT := COUNT - 1; 

if COUNT = 0 then - - the name is fully expanded 

if <a get request > then 

- - copy value from OBJ to appropriate field in message record; 
else 

- - copy value from appropriate field in message record to OBJ; 
end if; 

return; 

end if; 

case cfield name from message record> is 

when Fl => GETPUT(M, 0BJ.F1,C0UNT); 

when FN => GETPUT(M, OBJ.FS.COUNT); 

end case: 
end; 

Here T is a record type of an object being passed in. and F1..FN are the fields in the record type. If one of the fields, FI, say, were 
an access variable, that access variable would have been replaced by a record (as described in the pointer agent section above) 
and the action for the corresponding case would first check to see if the requested object were on the current site or elsewhere. If 
local, then the call to GETPUT would be made as shown above. If elsewhere, then an appropriate message would be propagated 
to the pointer agent on the indicated site. 
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Menas* Record Structure 

TKe intcrp r o c assor message structure ie key to the operation of the above object referencing s c heme . For each source package* 
a different message record type is defined. -These records consist of a fixed part and a variant part. There is one case of the 
variant part for each type of data object defined in the source package. In the case of a subprogram or task entry call* the 
variant part of the record contain! fields for all of the arguments, and if applicable, a function result. The fixed part of the record 
contains field selectors which are used for accessing fields of records, as described above. A simple example of % message record 
type b given below. It should be setfexplainatory feom the previous discussion. 

type MESS JT( DATATYPE: T.ENUM ) is 
record 

OBJ^ENUM : N-ENUM ; - * indicates outermost object 

TYPEl-FIELD : T,ENUM; - - record type TYPE1 
TYPE2JC1 : TYPE2JCUT; - - 2-dim array type TYPE2 
TYPE2JC2 : TYPE2-X2.T; 

case DATA-TYPE is - * reflects data to be exchanged 

when TYPE1-D => 

TYPEl-VAL : TYPEl; 
when TYPE2-D => 

TYPE2-VAL : TYPE2; 

when CALL1J) => - - function CALLl 

CALLl-ARGl : FLOAT; 

CALLl JtESULT : FLOAT; 

when CALL2.D => - - subprogram CALL2 

CALL2-ARGI : INTEGER; 

CALL2.ARG2 : INTEGER; 
when FLOATJ) => 

FLOAT.VAL : FLOAT; 
when INTEGERJ) => 

INTEGER.VAL : INTEGER; 
end case; 
end record; 
end; 

Since the postal service deals with all types of messages, a global message record type is defined. The global message record 
also consists of a fixed pari, and a variant part. The various cases of the variant part are, as one might guess, merely the different 
message records for each source package. The fixed part contains the destination package number, and the return address, which 
consists of the source site number, and a logical channel number. 

Translation Procedure 

The translations required for the methods outlined above involve numerous steps and are quite involved. In this section we 
describe briefly the procedures to be used and a utility that has been prepared to simplify use of the pre-translator. 

The first step in the translation procedure is to insure that the program to be distributed » correct. This is accomplished by 
compiling it for a single system. The programmer must do this before invoking the pre-translator. 

When a correct program is available, the translation and compilation procedure consists of the following steps: 1) determina- 
tion of the order of pretranslation of source files, 2) pretranslation of source files, 3) pre-link operations, 4) determination of the 
order of compilation of original sources (including agents) for target sites, 5) compiling and linking of individual site programs. 
Two utilities have been written to facilitate some of these steps. 

The pre-compilation utility (ADA UTIL) will translate the network of package dependencies implicit in a set of source files to 
a set of file dependencies in Unix “makefile* format. The list of relevant source files must be specified, and one or more targets 
(main programs) must be specified. Since the order of pre translation is identical to the order of Ada compilation, ADAUTIL 
takes an option specifying whether a makefile to run the pretranslator, or a makefile to run the Ada compiler is desired. 

The second utility, called MESSUTIL, performs step three above. The operations done during aetp 3 are: 1) constructing 
the global message record from all relevant package message records, 2) constructing a package of package site constants, 3) 
constructing main procedures for each site, and 4) constructing a xneta makefile capable of performing steps 4 and $ above. 

| Two scripts were written to simplify the pretranslation process. One script performs slops 1 to 3 above, and the other invokes 

the meta makefile to perform steps 4 and 5. If any non-Ada object modules need to be linked into any site, the met* makefile 
may be edited in between the running of the two scripts. 
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4 . DISCUSSION OF THE APPROACH 


One of our principal concerns with the system developed is the run-time overhead associated with the mechanisms we uard. 
We can model this performance in terms of the run-time overhead associated with various kinds of remote references. From 
the tests performed in [13[ we know that task rendezvous times exceed procedure call times by one and a half to two orders of 
magnitude, and that task elaboration times are several times larger than rendezvous times. We can also reasonably expert the 
network communications times to be sizable. For example message end-to-end times for MAP are on the order of 100ms, more or 
less independent of message size [14], for the Intel hypercube, a few milliseconds, and for the NCUBE hypercube, several hundred 
microseconds to a millisecond, where the latter two depend somewhat upon message size, the variable component of mwg e sine 
being 1-10 mkroseconds/byte [15}. We thus neglect aO local procedure and function call times, and model our overhead in tern* 
of the number of messages and rendezvous required. 

Thus, let t m and t, be the times to complete a message transfer and local rendezvous, respectively and let and n ; be 
the number of messages and local rendezvous required for a remote operation or type o. Then, the time to complete a remote 
operation is 

n f * h 

In these cases, we represent the overhead by the pair (n^,n?). 

Whenever there are task elaborations involved, we represent the number by E. It is listed separately since it is generally not 
necessary to do the task elaboration with each access, but only when tasks or procedures are first elaborated. Nevertheless, even 
though many of these need be done only once immediately after system load, the number of tasks in the system could have »•* 
impact on the scheduling algorithms to be used and the efficiency of any runtime system, and the number E is thus important 

The following sections present briefly the costs associated with each of the remote operations. 

Data Objects — (2,4), E = 0 

Access/ Updates to data objects require two messages and four rendezvous. One message is to send the request and the mc on- 
to receive an acknowledgement. The rendezvous are for the mail system. This presumes that the requested object is on c be fir*t 
remote site accessed. If there is a continuation to other sites through pointers, the above numbers must be mulit plied by ilie 
number of remote accesses required to satisfy the request. 

Task Objects — (2, 6), E = # of entries 

Task objects are accessed through entry calls. This requires two messages as for data objects and six rendezvous for »>nrhm- 
nization (4 for the mail system and 2 for the handler). 

The number of task elaborations that need to be done initially is equal to the number of entries to the task. Entry calk to 
task objects created from task types require no special handling by themselves. However, each task object created from a remote 
type require? two messages for creation and four rendezvous for synchronization. All further access are as in the rase of task 
objects. 

Procedures and Functions — (2, 6); E = 1 

Since the local agent treats procedure and function calls in the same way as task entry calls, the analysis is analogous. 
Pointers 

There are two factors to consider here, the overhead when the object pointed to is remote, and the overhead when the object 
is local. Remember that all pointers are replaced with records having a site number and a pointer. This requires that all accesses 
via pointers begin with a check of whether or not the object is local or remote. If remote, the time of the check will be insignificant 
in comparison to the time required for the remote access and may be neglected. In this case the overhead depends upon the type 
of objected being referenced, and will follow the results obtained abov*. 

However, if the access is local, the overhead is more significant. The exact amount of degradation will depend upon 
an individual compiler implements pointer accesses and if then else constructs. In a simple test in which we wrote as efficient 
assembly language code as we could for local pointer accesses with and without the pointer record construct used here, the 
difference was a factor of four. In interpreting this, however, one must take into account the magnitude of time involved \oniy 
a few microseconds are the most) and the frequency of occurence. With these considerations taken into account, we do not feet 
that much overall time will be added to local accesses. 

Summary Analysis Comments 

To place the above analyses in perspective, one must compare typical times for message transfers and rendezvous. Some 
typical network times were mentioned above. Rendezvous times on the order of 500-600 microseconds have been reported for an 
8 mHz IBM PC/AT. and on the order of 300-400 microseconds for Motorola 68000 processor. It Is ilso the case that these timr* 
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have been dropping significantly .-;th e«vJi ne^ release of Ada compilers intended for real-time applications, and are predicted 
by Ada vendors tc become ye< ron«id*rabIy smaller over the rext >ear or two. Thus, except for the fastest networks, the message 
times will either be close to the rendezvous times or dominate them, and the approach taken will be primarily influenced b> the 
network message time*. 

There is further issue that may be of concern, the number of tasks and GETPL’T routines needed in the local agents. These 
have a linear dependence upon the number oi entries (and subprograms) and types present in a remotely acee. ed package. While 
this may seem rather large, one is not likely to access a large number of things remotely, and those that are accessed remotely 
can be packaged separately from those that are not, thus keeping the number of extra tasks and routines to a minimum. 

5. STATUS AND CONCLUSIONS 

At the present time, the distributed translation system is operational for distributed packages with simple objects in their 
visible parts, i.e., no record or array definitions. Scalar data objects, subprograms and declared tasks may be directly referenced 
(no timed or contional calls). Tests have been successfully completed with up to three VAX processors cooperating on the 
execution of a single program. The implementation of the strategy described for referencing arrays and records (with fully 
concatenated names) is nearly complete, and expected to be in operation within a few weeks. 

Nevertheless, there is still considerable work to be accomplished before the distribution of library' packages and subprograms 
is complete. Although the strategy has been determined (see [16<), work has not yet been begun on handling timed/conditional 
task entry calls. Similarly, the dynamic creation of tasks is not complete. Two strategies will be implemented in this case. In 
the first, the created objects will be placed on the site elaborating the definition of the task type. In the second, the task object 
will be placed on the site creating the task through a declaration or new operator. The first is simpler to implement, but may 
make the task objects remote from the unit executing the code calling for their creation, while the second implementation is 
considerably more complex, and as noted in 9;, may contain hidden remote object references. Finally, task termination must be 
properly handled. 

More importantly, there are many issues of language definition that must be addressed. Our work has only addressed one 
point in the problem space to date, homogeneous, loosely coupled systems with static distribution. Additional representation 
mechanisms are needed to describe limitations dependent upon architectural considerations, to describe binding mechanisms, 
and to describe processor types (so that implicit data conversions can be accomplished). Moreover, it is probably necessary to 
require greater use of representational specifications on data objects to which remote access is allowed. Finally, there should be 
a more explicit definition of the allowed units of distribution. 
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Figure l: Loosely coupled system upon which we seek distributed program execution 
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Figure 2: Overall operation of translation system 
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Figure 3: Structure of translated example program 
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Abstract 

Tb b^apn* s4ehwe 4^&pmputational problem of Inverse km«n*tici and inverse dynamics of robot manipulators by 
taking advantage of parallelism and pipelining architectures^ For the computation of inverse kinematic position solu- 
tion, a maximum pipelined CORD1C architecture has been designed based on a functional decomposition of the closed- 
form joint equations* For the inverse dynamics computation, an efficient p-fold parallel algorithm to overcome the 
recurrence problem of the Newton-Euler equations of motion to achieve the time lower bound of Offlog-n]) has also 
been developed. 


1* Introduction 

Robot manipulators are highly nonlinear systems, and their motion control is usually specified in terms of the path 
traveled by the manipulator hand in Cartesian coordinates. To perform a simple kinematic path control, the controller 
is required to compute accurately the joint angles of the manipulator along the desired Cartesian path at an adequate 
and acceptable rate. To perform a dynamic path-tracking control, one must repeatedly compute the required general- 
ized forces, from an appropriate manipulator dynamics model, using the measured data of displacements and velocities 
of all the joints, and the accelerations computed from some justifiable formulae or approximations, to drive all the joint 
motors. In order to achieve fast convergence of the control algorithm, a sampling rate of no less than 60 Hz is prefer- 
able because the mechanical resonant frequency of most industrial manipulators is around 5—10 Hz. The above 
kinematic and dynamic path control reveals a basic characteristic and common problem in robotic manipulator control 
— intensive computations with a high level of data dependency. Despite their impressive speed, conventional general- 
purpose uniprocessor computers can not efficiently handle the kinematic and dynamic path control computations at the 
required computation rate because their architectures limit them to a mostly serial approach to computation, and there- 
fore limit their usefulness for robotic computational problems. This paper addresses these intensive robotic computa- 
tional problems by taking advantage of parallelism and pipelining architectures. 

Considering that most industrial robots have simple geometry, the kinematic path control requires the computa- 
tion of the solution of joint angles which can be obtained by various techniques. The inverse transform technique (l) 
yields a set of explicit, closed-form, non-iterative joint angle equations which involve multiplications, additions, square 
root, and transcendental function operations. Based on an actual implementation on a multiprocessor systemf [2,3) hav- 
ing a circuit to synchronize the CPUs and software scheduling for computing the joint solution, the best reported com- 
putation time was 3.6 ms for a six-link manipulator versus 20 ms running on a uniprocessor system. If we use a 
CORDIC (Coordinate Rotation Digital Computer) architecture [4], the computation time reduces to 40 ^z, a speed-up 
factor of 500tt* 

For the dynamic path-tracking control, there are a number of ways to compute the generalised forces/torques 
applied to the joint motors [5], among which the computation of joint torques from the Newton-Euler (NE) equations of 
motion is the most efficient and has been shown to possess the time lower bound of 0(n) running in uniprocessor com- 
puters [6,7], where n is the number of degrees of freedom (DOF) of the manipulator. Based on the study of Luh, Walker, 
and Paul [7j, it requires (150n - 48) multiplications and (131n — 48) additions per trajectory set point for a manipula- 
tor with rotary joints. It is unlikely that further substantial improvements in computational efficiency can be achieved, 
since the recursive NE equations are efficiently computing the minimum information needed to compute the generalized 
forces/torques: angular velocity, finer and angular acceleration, and joint forces and torques. For a Stanford robot arm 
(a total of 308 multiplications and 254 additions is required to compute the joint torques [8j), this amounts to 25 ms 
processing time on a uniprocessor system and 5.69 ms running on an experimental multiprocessor system with 7 proces- 
sors [9j. If we use the parallel algorithm with 6 processors as proposed in this paper, this reduces the computation from 
852 multiplications and 738 additions running on a uniprocessor to 197 multiplications and 183 additions for a PUMA 


This work WU supported in part by the National Science Foundation Engineering Research Center Grant CDRSS00022. Any opinions, findings, and 
conclusions or recommendations expressed in this articie are those of the author and do not necessarily reflect the views of the fuadiag agency, 
t The ms It i processor system consists of a M CSS 09 CPU and seven 2*0 CPUs. Each ZM is accompanied by two 9511 APUs, local memory, and 2/0 
interfaces. 

ft A speed-up factor is defined as the ratio of the compstational time of a task rosning on a uniprocessor system to the computational time of the 
same task running on the proposed architecture (i.e., 20 ms/ 40/1 s = 500). 
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robot (due to different kinematic structure of PUMA and Stanford robots, direct comparison on processing time is 
invalid) [10]. 

This paper discusses the development of a maximum pipelined CORDIC architecture for the computation of 
inverse kinematic position solution to achieve the pipelined time of 40 and an efficient p-fold parallel algorithm to 
achieve the time lower bound of computing the joint torques. The CORDIC architecture was designed based on a func- 
tional decomposition of the closed-form joint equations. Delay buffers are necessary to balance the pipelined CORDIC 
architecture vo achieve maximum pipelining. The buffer assignment problem is solved by the integer linear program- 
ming technique. The efficient p-fold parallel algorithm can be best described as consisting of p -parallel blocks with 
pipelined elements within each parallel block to achieve the time lower bound of O^log**]) of computing the joint 
torques based on the Newton-Euler equations of motion, where n is ihe number of degrees of freedom of the manipula- 
tor. The algorithm can be implemented with a group of microprocessors without complex intercommunication among 
processors and bussing of data. A modified inverse shuffle scheme is suggested for connecting the processors together 
with efficient intercommunications. 


2. Inverse Kinematic Position Computation 

The general kinematic problem of a ft- DOF robot arm concerns the problem of finding the generalised coordi- 
nates q = together with the vector of their generalised velocities and the vector of their generalised 

accelerations in the n- dimensional space such that the characteristics of the motion of the free end, the hand, coincide 
with the pre-$pecified Cartesian trajectory. This inverse problem has earned considerable attention because of its 
importance in relating the Cartesian trajectory of the hand to the corresponding joint-variable trajectory of the mani- 
pulator. This paper focuses only on the inverse kinematic position solution. 

In solving the inverse kinematic position problem, we are always interested in obtaining a closed-form solution (ue. 
an algebraic equation relating the given manipulator hand position and orientation to one of the unknown joint dis- 
placements), which yields all the possible solutions in a fixed computation time. Fortunately, most industrial robots 
have simple geometry and exhibit closed- form joint solution. Utilising the inverse transform technique [l], the joiat 
angle equations of a six-link manipulator with simple geometry reveal the computation of a large set of elementary 
operations: real number multiplications, additions, divisions, square roots, trigonometric functions and their inverse. 
However, these elementary operations, in general, cannot be efficiently computed in general-purpose uniprocessor com- 
puters. In order to obtain a fixed computation time for the joint angle solution, time-consuming transcendental func- 
tions (sine, cosine, and arc tangent) are implemented as table look-up at the expense of the solution accuracy. The 
CORDIC algorithms [11-14] are the natural candidates for efficiently computing these elementary operations. They 
represent an efficient way to compute a variety of functions related to coordinate transformations with iterative pro- 
cedures involving only shift-and-add operations at each step. Thus, cordic processing elements are extremely simple and 
quite compact to realise [14 J and the interconnection of CORDIC processors to exploit the great potential of pipelining 
and multiprocessing provides a cost-effective solution for computing the Inverse kinematic position solution. 


2.1. CORDIC Algorithms and Processors 

In conventional uniprocessor computers, computation of elementary functions such as square roots, sine, t cosine, 
hyperbolic sine and cosine and their inverse consumes a considerable amount of effort than multiplication operation. 
These elementary functions can be efficiently computed by the cordic algorithms which can be described by a single set 
of iterative equations parametrized by a quantity m ( = —1,0,1) which determines the type of rotations. To establish 
connections between CORDIC and rotation-based algorithms, let the angle of rotation 0 be decomposed into a sum of a 
sub-angles {d,; i = 0, n — 1} 

* = E (i) 

xM) 

where the sign u, (±1) is chosen based on the direction of rotation. Similarly, the plane rotation matrix R((?) 

m .L, I (2-*) 

or hyperbolic rotation matrix R(^) 
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sin*? 

-sin*? 
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cosh*? 

sinh? 

[-sinht? 

cosht? 


can also be decomposed into a product of sub-angle rotation matrices 

m = nRK) 


(2.bj 


( 3 ) 


Thus, a single rotation of Q angle can be replaced by n smaller rotations with d,- angle each. In the cordic algorithms, 
d, is chosen such that 


d, - 


Un-‘(2-<*>) , m = 1 (circular) 

2 -, l'* , m = 0 (linear) 

tanh - '(2 _ * ( ’>) , in - -1 (hyperbolic) 
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where m determines the type of rotations and {«(»); » * 0, n— 1} is a non-decreasing integer sequence. Using df from 
(4), R(d,) can be written as 




1 

1 


(S) 


where p, is a scaling factor and equals to (1 + n2‘ Sl l , *)'\ Let R*(f) and R w (d,) be the normalised form of 11(1) and 
R(d,) t respectively, then from (3), we have 

rw - n n ill »*«) - (••*) 

■ ■4 i* 4 ) #*0 


where 


; R s w - n 


1 

m, 2 1 


(•*) 


Usually, k m is a machine constant and = 0.6072 (for m — l) or 1-00 (for m =* 0) or 1.205 (for m * — l) t when n > If 
[12, IS]. The normalised rotation matrix of (6.b) indicates that each small rotation can be realised with one simple 
shift-and-add operation. Hence, the computation of a trigonometric function can be accomplished with n shift-and-add 
operations, which is comparable to conventional multiplications. This makes a CORDIC ALU a very appealing alterna- 
tive to the traditional ALU for implementing the elementary functions. In general, the normalised CORDIC algorithm 
can be written as follows: 

FOR t= 0,1, • • • , n-l,DO 




1 -m «, 2-< 1 > 


.fill. 

sac 

u, 2 _,<, l 1 

< 


(7.b) 


where z$ = xq, y% = y^ m determines the type of rotation, d, is chosen as in (4), and the auxiliary variable if* is intro- 
duced to accumulate the rotation after each iteration. And the corresponding “unnormalised” CORDIC algorithm is 
described as: 

FOR i= 0,1, • , n— 1, DO 




1 -m u, 2~* l '> 



1 , 

= P. 

«, 2-<‘> 1 




*+i = + *,4 (8J>) 

where x 0 - x 0 and y„ — y ir It can be shown that x, and z, v will accumulate the angle of the total rotation and have the 
same value after n iterations. However, the end results of (x,,*.) from the iterations of ( 8 .a) and the end results of 
(x^y*) from the iterations of (7.a) are related according to 

*. “ *m ; ». = 9m (*) 

Consequently, one may evaluate z; v and y* by using only tbe shift-and-add operations in ( 8 .a), then realise z. and y, by 
other simple methods such as ROM look-up tables and regular combinatorial logic, etc. Fortunately, it is possible to 
find a simple way to normalise tbe scale factor k* using the same shift-and-add hardware [14, 15]. The supplementary 
operations that are used to force the scale factor km to converge toward unity can be either performed after all the 
operations of (7.a) are terminated, that is, 

- (1 + 7 , 2 -) x* ; y / +1 - (1 + 7 , 2 -‘)rf ( 10 ) 

where x£ = x, v , y 5 = y. v , and 0 < 1 < a— 1 , or interleaved with the operations of (7 .a), that is, 

= (1 + % 2-) x? ; y? - (1 + 7. 2 -) y," («) 

where 0 < » < n-1. The parameter 7 ,' in (10) or (11) may be -1 or 0 or 1 depending on the value of » and the type of 

rotations (i.e. m) [14, 15]. 

Haviland et ai. [14] realised the CORDIC algorithm on a CMOS chip and showed that the processing time of the 
CORDIC chip is 40 /is. They also suggested n = 13 as the minimum cycle time of a two-byte (24-bit) fixed point opera- 
tion. However, in practice, they used n = 24. For a conventional CORDIC module, it requires 5 shift-and-add modules 
to compute one CORDIC iteration and one normalisation iteration in parallel (that is, 3 shift-and-add modules for (7_a) 

and (7.b), and 2 shift-and-add modules for (ll)). The desired output can be obtained in 24 iterations (n = 24). Thus, 

24 iterations of 5 shift-and-add modules computing in parallel will be enough to realise CORDIC algorithms. This indi- 
cates that the CORDIC processing time is no slower than the time for a serial multiplier computing two 24-bit operands. 
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Figure 1 muttriMi the dMMtUry fbctio— Uut cu U obtained (m Ut CORDIG ptoc— or vka us k Ml to 
-I* 0, or I. to this Kftrt.a CORDIC pr oces s or is depicted u&ba with tkm inputs ap » f* > *» whkh art tto toifttol 
▼alee* of s, aid x, to (•), u ««Q as Urn outputs tkat cormpoad to Um ftu) values of ^ t | a , ud a. to (t). Tto, 
Um outputs a. , f. , a. are the dtand ekaeitaij functions, vk«« m is appropriately set to -1, 0, «r I. These CORDIC 
processors will be coaf gated aad coutcUd, based os a functional dtcoapoatios of tbs jotot nagk eftltow, to atvisa 
at am eflkieat arckitectars for tbs computation of iavtm kinematic position solution. 

2.2. CORDIC Arcbitocturo for bvtrst Kinematic Position Computation 

Tbs desaga philosophy is to examine tbs inverse ktosmatk posit ioa solatioa for its computational flow aad data 
depeadeacies ia order to functionally decompose tbe compatatioas iato a set of CORDIC computational modules (OCMb) 
with aa objective that each COM viD be reaSaable by a CORDIC processor* This functional dacsmposition is nut 
aa sfa r aad can be best represented by a directed task graph wiik a laite set of nodes denoting Odds and a rnrtispmd 
tog laite set of communication edges denoting operands movements. An examination of tbs inverse ktosmatk position 
equations to Appendix A shows a limited amount of parallelism with a large amount of wqurntisimi to tbo flow of com- 
putations aad data dependencies. This serial nature of tbs computational flow lends itself to a pipelined CORDIC pro- 
cessors implementation [IS]. The decomposition of tbe inverse kinematic position equations is duns by looking at Um 
equations which can be computed aa elementary functions by CORDIC processors as Gated to Figure 1. 

The task of computing the inverse kinematic position of a PUMA robot arm, based on tbe equations to Appendix 
A, can be decomposed into 25 snbtasks to which each subtask corresponds to a COM aad can be realised by a CORDIC 
processor. For example, # x can be computed by the following 3 sub tasks: 


Sub task 1: *1. - r — (p * + 



CORDIC Processor: CIRC 2 


t 

*0“P. 
»o - ~r, 
*0 “ ® 


Sabtuk 2: *2. - V'* - <1 - V*»2 “ *1 


CORDIC ProceMor: 


HYPE 2 


f 

*o-0 


Subtask 3: 


* 3 . 7 *i - * 1 . - 


CORDIC ProctMor: CIRC 2 


* 8 - * 2 . 
Il — 
* 9 - 11 , 


The computational flow of these 25 tasks together with the input data can be represented by tbe directed acyclic data 
dependency graph (ADDG) with switching nodes aad parallel edges as shown to Figure 2 and tbe details about tbe 
decomposition of the inverse kinematic position solution into CCMs can be found to [4]. In Figure 2, each computational 
node, indicated by a circle, represents a CORDIC computational module, and each switching node, indicated by a dot, 
perforins no computations but just switches data to various CCMs. The operands or data move along the edges. A 
major bottleneck in achieving maximum throughput or maximum pipelining to Figure 2 ia the different arrival time of 
the input data at the multi-input CCMs (e.g. nodes T18 and T22 ia Figure 2). The computatioas of multi-input CCMs 
can not be initiated until all Um input data have arrived. This different arrival time of input data leagtbens the pipe- 
lined time. Thus, the ADDG is said to be unbalanced aad fails to achieve maximum pipelining. Several teckaiques [l7j- 
[19] have been suggested to remedy this data arrival problem by imserttog appropriate a umber of buffers (or delays) to 
some of the paths from the input node z to the multi-input CCMs to “balance” the ADDG and achieve maximum pipe- 
lining. This buffer assignment problem for balancing the ADDG can be reduced to aa integer linear optimisation prob- 
lem. Detailed formulation of the optimal buffer assignment problem as an integer linear optimisation problem can bu 
found in [4]. After solving the buffer assignment problem, realisation of the balanced ADDG results to a maximum pipe- 
lined CORDIC architecture. For a PUMA robot arm, the architecture consists of 25 CORDIC processors and 141 buffur 
stages with 4 tapped- delay- line- buffers [4). The initial time delay of the pipeline is equal to 18 stage latency (or 
720 ps)» where the stage latency of a CORDIC processor is assumed to be 40 p* [14]. The pipelined time of tbe 
CORDIC architecture equals to one stage latency or 40 pa. The realisation of the maximum pipelined CORDIC archi- 
tecture is shown in Figure 3. 
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S. Invtm Py uimWi Coa p ^ Utt o a 

Tk gftml iamtt ijiaak problem lor u «-Bak ouipvUtor cam bo stated mo follow: itmlkjimt positfe— 
and velocities , f,(t) ) "_| which describe the state of tbe nuiptbtor mi time f , together with tbe jsint accelera- 

tions (£(«)}•», which are desired mi the poiat, solve the djumk equtiort of motion for the joimt toryii {»>(!)}*-* M 
follows: 

(u) 

wfc*r« f(l) - Jr„ * fc ...,r.) r , q(t) - , q(t) - (f „ ,*.) r , 3<) “ ft’i. f»-~4.| r . H ) * ** •*! 

vector function mmd »« prescript T demotes transpose operatioe om matrices mmd vectors. 

At preseat, mmch atteatiom has bee a hewed om the coespetatiomaJ issees of the inverse dynamics based om the 
Newton-Euler (NE) formulation, resulting ia various meltiprocessor-hased coatrol syslenw |t ,21-24]. The le c ur sr re strmc* 
tare of the NE equations of motion is obviously well smited to standard mngk-instr met ion-stream and single-da ta-stream 
(SISD) computers. It is, however, act aa eficieat parallel proressiag for acw aigh instruction-str ea m mmd muktipk- 
d a ta-stream (SD4D) com paters that are capable of performimg many simultaneous operations. Oar app r oa c h in design- 
tag eficieat algorithms for compatiag the robot ia verse dynamics is to look at the compete tional complexity of the 
problem Irst. Ia partkelar, we aeed to kaow what is the Kmitatioa of speediag ap the computation ef the ia verse 
dynamics while fanning oa p processors, where l < p < n. That is, we would like to establish a time lower boend for 
the inverse dynamics comp station problem so that several eficieat compatational schemes can he compared and con- 
trasted. Them eficieat algorithms achieving the time lower boaad caa be designed for the competation of the inverst 
dynamics. The following notations and lemma will be ased to derive the time lower boaad of the inverse dynamic prob- 
lem. 

Notations: 

(1) Lower ar ithme tic expression is any well-formed string composed of foar arithmetic operators (+,— ,X,/) or, for 

convenience, two operators +(or -), X (or /), left and right parentheses, and atoms, which ars constants or vari- 
ables. We denote a linear arithmetic expression E of m distinct atoms by E<m>, e.g. E<4> :•+!-« / d. 

(2) T, |/,(*) , /j(*) , ... , /.(•)) » Miiimim competing time needed to evaluate [ /,(•) , /,(•) , ... , /,(•) ) 

using p processors. 

Lemma 1: The time lower bound of T p (f?<m>] [25j. The shortest parallel time to evalaate a linear arithmetic 
expression E<m> using p processors is bounded below by and equal to 0{\m/p] + flog 2 p]), that is, 

T , (E<m> ] > 0\\m/p} + [log 3 pi) 

Theorem 1: The shortest parallel time to evaluate the joiat torques { r ; (0) ^ equation (12) using p processors 

is bounded below by 0(k t fa/p] + k^ flog a p]), where k x and k 7 are specified constants, that is, 

r,[r„rj,...,r,] > Ofk, fi»/^l + **flo*jpl) (IJ) 

The proof of Theorem 1 can be found in [10]. Two ext** me cases follow from Theorem 1: 

(a) If p « l, then the shortest computing time T f [?!. ?*...,?»] is not lower than 0(a). Thus, the NE formulation is the 
most efficient algorithm of evaluating the inverse dynamics running ia uniprocessor computers. 

(b) If p = », then the shortest parallel computing time T p [fit 1 *— » r «] is not lower than Offlog^]). 

Theorem 1 indicates that an efficient algorithm running on p processors may not achieve the same time order as 
0(k, [*/p] + k,flog 3 p]). However, if a parallel algorithm possesses the time lower bound, then it mast be the most 
efficient algorithm of evaluating the inverse dynamics. Theorem 1 also indicates that, although NB formulation is very 
efficient for computing the inverse dynamics, a better solution is to find an efficient parallel algorithm, ramming on p pro- 
cessors, that possemes a time order of 0(k x [n/p] + k* [log* p]). A parallel algorithm ruaning on an SIMD machine and 
achieving the time lower bound is discussed next. 

The recursive NE equations of motion are very efficient in evaluating the inverse dynamics whether they are for- 
mulated in the base coordinate frame [6] or the link coordinate frames [7]. The clear advantage of referencing both the 
dynamics and kinematics to the link coordinates is to obviate a great deal of coordinate transformations and to allow 
the link inertia tensor to be fixed In each link coordinate frame, which results in a much faster computation ia a unipro- 
cessor computer. However, the recursive structure of this formulation is in an in homogeneous linear recursive form, e.g~ 
w, * a, w,_, + i lf where a, * (n 3x3 rotation matrix) and i, ~ *oq,t» which requires more calculations and 

arrangements for parallel processing than the homogeneous linear recursive form. On the other hand, the NE formula- 
tion im the base coordinates caa be re-arranged and transformed into a homogeneous linear recurrence form, e-g. 
w, * + f t a,—,, which is more suitable for parallel processing on aa SIMD computer, yielding a much shorter com- 

puting time. 

Once the NB equations of motion are formulated in the base coordinates in a homogeneous linear recurrence form, 
then a parallel algorithm, called recursive doubling [16,1 7,24-27), can be utilised to compute the kinematics in the for- 
ward equations and the dynamics (or torques) in the backward equations [5]. The homogeneous linear recurrence prob- 
lem of site (*+l) can be described as follows: given x(0) - «(0) * identity and «(*), 1 < » < a, fiad x(l)* x(2), ... ,*(*) 
by an algorithm running on an SIMD computer of n processors, where 

t n, u4 1; uf im 4 k«N m vktiabM. 
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«(.) - *{.-!) * .(0 ( M ) 

m0m Imtot u tMociaUft eytn tof, vkkJi eu be a malm pm 4a ct or idfitlw, a rector dot p roduct or s ddili m, 
•tc. If «(0) ii tyil to aa ideatii y, then the linear recurrence of sis* (*+l) caa be redact to tbe case of om * by lab 
ia| a itiA operation as follows: «(0)«-a(l) t •(•-!)♦-•(»)» where V" denotes a replacing opera 

tioa. Tie r e cu r sive doubling algorithm bask ally solves tbe homogeneous Sam recurrence fora ia (14) by ipfiUiat tbe 
computations of a serial associative opera tioa. For aa arbitrary *(a)^ tbe generalisation of tbe iba aloes (a-fiyi 

parallel operations at tbe &nt splitting, (a + 1 )/4 at tbe ac o t d, and (a+lj/l 1 at tbe k tb mattl [bfjail)] spSUb p L 

tbea a (a) ia csaysUd with oac laal operation. Similarly, it caa be mo w n tkat aft) caa be rnapilid ia [lo| 9 (i‘fl| 
ipfittbp, where 1 < t < a. la other words, with tVe recursive doobfing algorithm, x(l), *(2), ... , x(a) caa be comp ete d 
concurrently ao later tbaa tbe time step fkgjs + l)]. 

Tbe proc e d ure ia cosapotiag the inverse dynamics from tbe NB epit ioe s of motion formalated ia tbe base coe ad t 
mates ia to re-arrange tbe kiaematk epalioM atd tbe dyitaic agnations ia a bomo gtat oa a linear a cswl t t Iona, aad 
tbe following mpot paraaaeters relating tbe fbnunlatioa moat be given or evaluated in advance: 

(a) Tbe 3x3 rotation matrices '**0,, t » 1,2,...,*, mrbkb abates tbe orientation of Salt s' coordinates ref erence d In 
fink (a — l) coordinates, need to be evaluated ia advance. 

(b) *p* denotes tke origin of Bak t coordinate frame from tbe origin of Eak (a— 1) coordinate frame ex p res s e d wilb 
respect to Bak t coordinates; f s, denotes tke location of tbe center of mam of Bak « from tke origin of Bak t coor- 
dinate fraac expressed with respect to link i coordinates; aad * J, denotes tbe inertia matrix of Bak t a boot its 
center of mam expressed witk respect to Eak s’ coordinates, mast be given ia advance. Note that *p, , *a,-, aad *J,- 
are coastaats wbea referred to tbeir own link coordinates. 

(c) X, is a joint indicator wkick specifies link s’ is rotational or traariatioaal as follow: 

| 0 , if Eak s’ is rotational 
* \ 1 , if link s’ is translational 

(d) Let — 0 f pQ « \§ a9 p yv f g ] T aad [g| m IJQIllsi/s 1 . If external force f, and external smombI d, are 

exerting on Bak a, tbea f i+ , » f„ n t ,| * n 9 ; otherwise, f t , ( * n^t — 0. 

Tbe procedsre of evaluating tbe NB equations of motion as a linear recar re nee problem is then given below (note 1, ace 
need here as variables): 

STEP 1. Compete tbe rotation matrix °R, with respect to the base coordinates for i * 

•R, (») 


STEP 3. Compete p,* , e t> and s, for s’ — 1,2, ,a 

• *0 . *0 * (o«° 4 l r ; Pi'* °S, ‘Pi* ; - °R, ■«, 

Tbe evaluation of s, only involves taking tbe third column of °R,. 

STEP 3. Compete 

*. - (i - M 

and 

- *i-i + *• 

STEP 4. Compete 

“ (*•— I fi I X l|_i fj) (l “ ) 

and 

STEP S. Compete 

», - «. X p,* + *#, X (w, X p/) + (•<_, f { + * •»,- X (•,_» *, )) \ 

and 

?.• - p.-i + *. 

STEP fi. Compete 

(«, X a,-) + Pi 

STEP 7. CompaU 

r, - •,?, 

STEP t. CompaU 


d») 


to) 

(«) 

d*) 

W 

w 

(a) 

ta) 


N< - J, «. + w, X (J. *»,) 
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w 

t») 



tkmt tkc 


For tbt nfa of nviac tW cakilitioM of oralutiag J 4 » X *J 4 *R» wkick Lmk «t aL (7] ik i w f 
cmjmUUm m *ut« coaopBcotcJ, (») ■ bmUSmI to 

'•». - ‘*o •». - (**,) r •».- J {•K,) r 


STEP 1 Cwpite . 
STEP 18. CoaptU 


‘N. - *J 4 + •«, X f J 4 '«.) ; N 4 - •*, -N, 


fi - f 4+I + F* 


f 4 « n 4 + (p 4 + %) X f 4 + p 4 X fi>i 


(») 

(«) 

(») 

(») 


STEP 11. CoafiU 


■i - “•>« + *. 




( n i) r »i-i » if\-0 


(30) 


(31) 


Pnrioiilj vukiftd Um» l expressed la tkc beet coordinates, arc jpvo as follows: n\ is the mass of Ink * f «,• Is tkc 
angular Telocity of link «, w, is tkc asgilar acceleration of fink t, p, is tkc linear acceleration of link t, r, is the linear 
acceleration of tke center of mass of link i, F,- is tke total force exerted on link i at tke center of mass, N,* is tke total 
moment exerted on fink t at tke center of mass, I* is tke force exerted on fink i by fink t— 1, n,- is tke moment exerted 
on link i by fink »-l, t g is tke torque exerted by tke actnator at joint i if rotational, force if translational, f t * is tke 
joint Tariable of joint i (#,• if rotational and if if translational). 

Bqnation (1$) skows tkat tke eTalnation of °R, is a simple recursive matrix product form. Equations (18), (20), 
(22), (22), and (30) only inrolre simple recursive rector addition form. Tke other eqnations in tke NE eqnations can be 
competed parallelly. Thus, tke eralnation of tke total coropatational complexity of tke parallel algorithm for n PUMA 
robot arm can be derived as follows: 

(a) Tke parallel eralnation of (1$) using reenrrire doubling indicates (27 [log}*] — 19) scalar maltiplkations and 
(18 riofan] — 14) scalar additions. 

(b) Eqnations (18), (20), (22), (28), and (30) all bare tke same recursive rector addition form, the total parallel evalua- 
tion of these eqnations requires (6 + 9 flog^n+l)]) scalar additions. 

(c) Tke parallel eralnation of tke other eqnations in tke NE formnlation, e-g. °R, ' p,-* f s t - = °R, F,*, N,* t r i9 

and all tke h g of (17), (19), (21), and (29) can be calculated by simple parallel computations, yielding a constant 
computation of 135 scalar multiplications and 98 scalar additions. 

Combining tke results of (a), (b), and (c), tke total computational complexity of tke parallel algorithm applied to a 
PUMA robot arm is (27 flog}*] + 116) scalar multiplications and (24 [logjrt] + 9 flog^n + l)] + #4) scalar additions. 
Note tkat it is of time order Offfog}*]) because we are using p =* n processors. If further reduction on tke coefficients 
of ( flog2«1) ■ desirable, this can be accomplished by using matrix multiplier chips. This would reduce tke coefficients 
27 and 18 in evaluating (15) as discussed in (a). If m ■* 6, then the complexity of the parallel NE algoritkm is 197 multi- 
plies (malts) and 183 additions (adds) as compared with the complexity of the NE algoritkm running on a uniprocessor 
[7]: 852 mults and 738 adds. Moreover, even if • becomes large, say n — 12 (for redundant robots), then tke number of 
mnltiplications and additions increases only by 27 and 33, respectively. Thus, we have shown that considerable savings 
in computation time can be achieved from embedding the inverse dynamic computation in a parallel algorithm, which 
has a time complexity of logarithmic in the number of joints, 0( [log} *])- 

td. An Efficient Parallel Algorithm With p-Fold Parallelism 

Last section showed that the bottleneck of parallel computation of the inverse dynamics depends on solving the 
homogeneous finest recurrence of tke N-E formulation. If tke restriction tkat one microprocessor “handles” one joint is 
relaxed, it is desirable to obtain an efficient parallel algorithm which can greatly improve the evaluation of the linear 
recur re ace using p processors. A parallel algoritkm of evaluating tke inverse dynamics with a restricted number of p 
processors has been developed to achieve the time lower bound of 0(k t [*/pl + k* [log} pi). The proposed p-fold paral- 
lel algorithm can be best described as consisting of p-paraDel blocks with pipelined elements within each parallel block. 
Tke results from the computations in the p blocks form a new homogeneous linear recurrence of sire p , whi ch again can 
be computed using the recursive doubling algorithm. Tke parallel algoritkm with p-fold parallelism (PFP) is summar- 
ised and presented as follows: 

Algorithm PFP (p-fold Parallelism). This algoritkm divides tke computations into p -parallel blocks of computations. 
The j th processor computes the elements ia the / th block serially. Tke results from the the p -parallel blocks form a 
new homogeneous linear recurrence of sue p, which can be computed by the recursive doubling algorithm. 
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Pi. [hitUKtttioaj Givta «(0) + ideality, kt<« [(n+l)/y), m ~ ye, mad eel 

( 

j «(i-l) , 1 < » < u+i 


M0- 


Jo , u+1 < t < m 

wktw e indicates ike block da (member of elemeate ia a block). 


PS. [Diride iato y blocks^ Divide M*)» 1 < » < at, imlo p blocks me follows; 

ytb block - (M(y-1> + 1). M(i-*)« + 2), amd let N(l J) - M((j-1). + 1) 

where N[iJ) iadksteo tbe t th element ia the jib block. 


*</<* 


PS* [Compete N[iJ) ia a DO loopj Tbe /th processor serially competes N(»j) ia tbe /th block as: 
For • » 2 step 1 matil e Do 


w 


N[ij) - Nii-ij ) • M(y-i) «+o . i<y<p m 

END 

It is seem that s(l), r(l), ... , x(e— 1) has bees evaluated ia the DO loop as well as M*4l 2 < » < s. That k 
*(l) - N[ 24). *(2) - N( 34). ~ , *(«-!) - ^4 

Pd. (Form a aew homo g eneous linear recurrence of sise y.) Let Tf;) — N(#J) and y(j) » *(>-1), for 1 < / < p and 
referring to (14), (32), aad (33), we have 

* W * *0-1) - *(l/-i> - 1) * «((i-0) * •((/- 1> + 1) * ... * •(>-!) (34) 


- y(/-l) • MO-O + 1) * M(/»l> + 2) * ... * Af{» - y (j-1) * N(*,/) 

Equation (34) is a aew homogeneous linear recurrence of rise y which can be parallel}/ evaluated by tbe algorithm 
FOHRA, running in time proportional to 0( [log^ yl) and yielding tbe resulU x(«-l) « y(l), x(2s- 1) ** y(2), 
*(y«“l) - I (y)* (Note that if (*+l) is divisible by y, then x(n) - y(y); otherwise, *(ys-l) * y(p) » 0). 

P5. [Compete intermediate s(i) in equation (35).] Without loss of generality, assuming that (*+l) is not divisible by y, 
then there are intermediate terms of z(») that need to be determined. They are: 

*0+0 » C < » < s— 2, 1 < / < y-2 and *((y— 0+0 t 0 < » < n-(p— l)s (25) 

Referring to (7), (33), (34), and (35), thereby giving 

*0+0 - *0-1) * «0) * *0+0 # ... * *0+0 - y(j) # M>+0 * M>+0 0 ... * M>+*+i) (*) 

- y(i) 0 N(t+i j) , o < i < s-2/ < / < p - 2 


and *((y- !)• + 0 — t(P“l) * M*+ 1#— 1) . 0 < i < u-(y-l)s 

where //(i+lj), y(jf) of (34) have been evaluated in steps PS and P4, respectively. Equation (34) shows that if 
(w— y— f+3) ta sk s are of tbe same evaluation, then the calculations of these equal tasks are suited to an SIMD com- 
puter of y processors. It is shown that parallel evaluation of (34) requires r(n-y-s+3)/yl time steps (note that ? 
(»+l) is divisible by y, then x(n) can be evaluated in step P4, yielding (a-y-s+2) equal tasks in (34), thereby 
requiring [(u-y— s+2)/y] time steps). 

END PFP 

U b seen that tbe total parallel computing time T p of the homogeneous linear recurrence of rise (s+l) usi n g p 
processors is: 

f rO+l)/>l+ r 0— p— *+3)/yl+ Tlosa Pi - 1» if (*+l) in not divisible by y. 

Tp " lf(*+I)/jl+f(*-y-«+2)/yl+ Tioga y] - 1, if (*+l)is divisible by y. 

Applying the above y.fold parallel algorithm to the N-B formulation for an *.fink rotary manipulator, it is able to 
achieve the time lower bound of 0{k x [+/p\ + k^flog, yl). 

The above n-fold parallel algorithm is suited to he run on an SIMD computer. A cascade structure can he umd 
for connecting the PEs. An alternate structure is to position a network between the processors and memories. The 
interconnection pattern, called the “perfect shuffle (24, 27],“ has the number of links between processors proportional to 
*. An attractive interconnection pattern, called “inverse perfect shuffle (26*27]," is suitable for tbe implementation ef 
solving the homogeneous linear recurrence aad can be obtained by reversing the arrows of tbe perfect shuffle. Detail 
about this network connection for solving tbe homogeneous linear recurrence for computing the joint torques can be 
found in [10]. 


4. Conclusion 

A maximum pipelined CORDIC architecture for computing the inverse kinematic position solution and an efficient 
parallel algorithm for computing the joint torques have been discussed. To achieve maximum throughput, delay btim 
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an to baluct tW pipeline. For a PUMA robot aim, tk CORDIC arddUctaic conmta o f 2$ CORDIC proces- 

■on aoi 141 Mtr stages with 4 tappa^elay-fiM kina. Tk initial tint delay of the pipeline is cqaal to 720 p* aad 
Iks fipdM tins of Iks CORDIC aicUttctan ego a It to oaa stage lateacy or 40 ps. The paialkl alpnUs lor coo 
pvtiat Iks Joint t o r q ots ku & time c omp l exi t y of io|ariUune in the amber of joints. The interconnection networks for 
tko p roces so rs ban aln boon investigated to improve tko itSHuUwi of commeakation ud internal btltnif betwcei 
prow ora in an SD4D compnter. Ubkg tko concopts of tko p ropo s ed parallel algorithm, it woald bo powilh to device a 
VLSI chip capable of implementing the inv ers e dyaanks compoUtion at speed priaianly boondod by tko pro p on e d 


b. Appmdb At bmm Kinematic Position Rotation 

Tbo hnm kinematic position probity can be stated ac Gina the poeitioa/oricatatioa of the maaipolator hand 
and the Hak/jomt pare meters, determine tbs joint angles so that the maaipolator can bo positioned so dobed. That is, 
g i r en 

■* n «. L 
_ *S S *1 7| 

*» •, «, 9* 

0 0 0 1 


n o a p 
0 0 0 1 


tbo joiat angle equations are: 

r-W + ftY* 


Un^l 


K 

- tan- 1 

** 

l* J 


±Vr*-di 


/n* " PmC 1 + f 9 S 1 ; film ** •*^1 + *>^1 > /«f* — F*^I + > f 13# - — **^1 + O f C| 

* * /n, + /nr - " •* “ •* i « - 


tan 


-1^1. 

^4 


— tan" 


±Ve -d* 

*r(/ — PjCJ + d 4 /ny — 


•a(/nr^a + P*^a) + *3/11* + rf aP* 


» — tan' 


-1 


(P*^3 “ /n^j) + (— ~)/iir + (~)P* 

4 J *3 


(p,Cj + /u,Sj) + (~)/nr + (~)p* 

•a *i 


P* ~ P» ~ fa 

— tan 


-S,a, + C l a y 


CAC A + *iS) " $»•» 


tan* 1 


/ 13# 


*?»/ 1U “ ^W*» 


tan“* 


gaK^Ci*, + V,) - + <?isl 


^ut^io, + 5,0,) + C„s, 


-tan“ l 


Ca(C 0 / He " ^33*t) + ^a/ 134 


^»/lU + ^23®* 


tan 


-CJgfe/n, ~ Sa».) 1 SJ iU ) ± *»($«/„. + C^o,) } 


SJPnf n» - ^s»i) + ^«/u# 


when (— — ) , (— ) , (— ) an constants, C f * con 5 f w sin C $ bcm (8 { + #,•), and Sy a sin (*,. + #,-). 
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A Run-Time Control Architecture for the JPL Telerobot 
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1. Atetract 


<jt ■ci"" lint ^an irchltactur# for lapXtMntlno th# proctta-Xtval decision Mklng for 
m bitrtrddcaXXy atructurad taltrobot currently balng iapXmntad at tht Jet Propulsion j 
Laboratory t JPL) ^Constraints on tha architecture dttign, architect ure partitioning concepts. j 
detailed description of th* f existing and proposed lapleaentatlona are provided. 

A - A- : * *■ - 


2. Introduction 



/ 


CL*. 


The architecture of a telerobot le required to support autonoaoue and teleoperated 
actlvitiee in a Banner designed to obtain the aaxlaua synergy of function between robotlce 
and teleoperation. The JPL telerobot lepleaente auch a ayatea for applications involving 
various servicing and repair operations In apace. 

The architecture propoeed for the JPL telerobot decoapoeee the functionality of syatea 
operations into three hierarchical levels. For the autonoaoue component of the architecture, 
theee level* conaiet of the Task Planning Level at the top of the hierarchy, followed by the 
Process- Level at the lntereedlate level, and the Actuation _and Sensor Level at the bottoi of 
■the hierarchy. Each of these levels is designed to operate robustly by using local feedback 
to detect and recover froe local error*. 


The Task Planning level Is concerned with the overall context of the taek. including global 
planning, overall execution eonitorlng and task replanning. The basis for the decision making 
Is the Task Sequence Logic vlth relatively little Incorporation of Task Doaaln Physics. As a 
consequence, activity tlelng at this level la driven by taek scheduling requlreaents. and the 
decision making methods used are predominantly symbolic and not numeric. In the JPL telerobot 
the functions at this level are mechanized by an Artificial Intelligence Planner subsystem 
<AXP> vlth the logical planning of a Module swap-out sequence being a representative example 
of internal subsystem activity. 

The Procees Level la concerned vlth the planning, execution and monitoring of subtasks such 
as grasping objects, object seeeably etc. Actlvitiee at this level require only a local 
subtaak context vithin which various eeneor and actuation subsystems are commanded and 
coordinated to accomplish a given eubtaek. Decialon aaking at this level hae to be cognizant 
of the phyeice and geometry of the taek doaaln and the occurrence of various run-tise 
physical events In the telerobot and lta environment. Decision aaking la characterized by a 
hybrid of aymbollc and numeric processing with activity timing determined by the event line 
of physical sensed events. Mechanization at this level la provided by a Run-Tiee Control 
subsystem CRTC) vhlch performs the various trajectory determinations, reflex specif lea t lone 
etc. for subtasks such as grasp centering and activation, parts sating etc. 

The Actuation and Sensor Level serves to provide the basic manipulation and sensor 
capabilities of the robot. The activity at this level is Intimately tied to the phyeice of 
the domain but no task context is required. Activity occurs in continuous tlae (practically 
realized by high rate servo sampling) and is predominantly numeric in nature. Functions at 
this level are ■echanized by the Manipulation and Control Mechanization aubeyetee (MCH) and 
the Sensing and Perception subeystea (SAP) with compliant motion execution or object tracking 
constituting representative examples of subsystem activity. 

* Similar levels may be Identified with the Huean Operator, with High Level cognitive and 
planning functions at the top of the hierarchy, motor skill coordination at the intermediate 
level and autcultr and sensory systems at the lowest level. 

Coordination between the autonomous levels of the hierarchy ahd the Human Operator ia 
provided by three methods. At the lowest level coordination la enabled by providing the 
Operator with the ability to directly Interface with the robot arm and aeneore via force 
reflecting Joysticks and stereo camera systems. At the top of the hierarchy the coordination 
Is provided via interactive task planning activity between the Task Planner and the Operator. 
At the intermediate level the coordination ia achieved via a mechanism known am Traded/Shared 
control described further below. 


Traded control refers to the situation where the Operator relinquishes control ower process 
level operations for certain aegeente of the task sequence execution while retaining it for 
others. The transition between the operator controlled segments and autonomous program 
controlled segments le achieved via a 'hand-off* protocol that ensures that information about 
the vorld state is correctly communicated between the operator and the machine. 
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Shared control refers to the situation where the Operator and the autonosous progress 
Jointly participate In decision making at the process level. The contribution of the 
autonosous programs could be passive In nature ms In the form of monitoring operator actions, 
e. g. , ensuring that Joystick actions will not lead to possible collision with the 
environment. The participation may also be much more active with the autonomous programs 
actually contributing to the - manipulation of objects In the environment, e. g. • letting the 
operator control the wrist of the robot arm while the autonomous programs control Its global 
positioning. 

The overall structure of such a system architecture Is depicted In Figure 1. 

3. Run-Time Control Architecture Requirements and Constraints 

Ve now focus on the architecture for the Run-Time controller. The functions of this 
architecture are to: 

1) Mechanize Process Level Operations 

2) Permit Han-Hachlne Coordination 

The specific design of this architecture Is influenced by the context within which the 
telerobot operates, the details of which are described below x 

1 ) Han-ln-Loop 

This feature has the most significant Impact on the design of an RTC architecture. The 
existence of the capability of teleoperation Implies that the autonomous capabilities of the 
autonomous RTC architecture need not be complete. It is sufficient to have an architecture 
that is capable of performing autonomously for a reasonably high percentage of the times the 
task Is required. Since most computational algorithms to plan and mechanize simple physical 
subtasks such as grasps etc. are exponentially hard to perform, this feature of the 
architecture enables algorithms to be selected based on their likelihood of successful 
performance. 

The architecture Is also required to functionally Interface to both the Operator and the 
AIP planner. This Implies that an Interpreter style command language is required to enable 
the human to effectively comeunicate with the system. The Operator must also be able to 
interact with the system in a manner that does not Involve a knowledge of the detailed 
implementation and computational aspects of the system. This requirement is usually taken to 
mean the Implementation of a Task Oriented Language where actions are implicitly specified in 
terms of the required task. This concept is Implemented in the RTC via a Task Command 
Language augmented by a constraint specification scheme. The constraint specification 
language not only allows the operator (or the AIP) to designate the required physical task, 
but also allows the specification of a set of constraints that are to be naintalned by the 
autonomous system during the execution of such a task. The autonomous system then determines 
the appropriate actions to execute the task and also satisfy the constraints. A sample entry 
of the RTC command dictionary is shown in Figure 2. The operator is also required to 
interact with the system in a shared mode of operation as described earlier. The constraints 
specification command language allows a natural and easy extension to permit the 
specification of shared control activity. 

2) Specialized Environment 

The telerobot is required to operate in a space environment performing taskm much a • 
satellite repair, structure assembly etc. The specialized nature of the tasks implies that 
standard factory automation paradigms are inapplicable. Further, unlike a highly changing 
manufacturing environment, the space application environment does have a substantial body of 
off-line knowledge available in the form of actual astronaut experience, earth simulators 
etc. The existence of this knowledge indicates that script based techniques are feasible for 
decision making, because they permit the transfer of the available knowledge to the systea 
with the least effort. Of course, as the telerobot takes over more autonomous functions, the 
scripts could be replaced with modules capable of reasoning and generating their own 
activity. 

The other constraint of the specialized environment Is the perspective shaped by the 
fact that the Run-Time Controller will eventually fora part of an Embedded systea Implemented 
in space quallflable hardware/sof tware. Also, since the controller is part of an end-to-end 
telerobot system, requirements of overall system design affect the design of the local 
subsystem architecture. While these constraints are not rigid since the first 

implementations of the JPL telerobot are necessarily ground based research prototypes, the 
requirements for an evolutionary growth in the system require that these factors be 
considered while designing and building the baseline architecture. 

3) Parallel Activity Tracks 

A requirement of the telerobot architecture Is the management of parallel task execution 
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and the simultaneous use of aany physical resources available to the robot. Physical 
resources consist of various manipulation resources such as robot arms, end effectors, mobile 
platforms etc. , or various sensor resources. Each of these resource could be active 
simultaneously as In the example of dual arm Independent or coordinated actions. The 
architecture must be embedded with appropriate resource logic to ensure that resource 
conflicts do not occur vhen simultaneous activity tracks are In progress. Further, the error 
detection and recovery for each of these Independently controlled resources must be 
coordinated. 

4 > Computational Bottlenecks 

Computational resources must also be managed. Host robotics algorithms are 
computationally very Intensive, far outstripping the capabilities of uniprocessor machines. 
Concurrent processing In a distributed multiprocessor environment Is necessitated, and the 
system should therefore be capable of performing the dynamic computation task allocation 
necessary to optimally use the available computational resources available to the system. 

In the BTC, the computational paradigm that has been adopted Is the message passing 
multiprocessing system. 

5 ) Robust Operations 

Robust operations at the any level of the hierarchy require the laplementatlon of 
feedback, where the actual progress of a subtask execution In the world is monitored to guide 
further actions. At the sensor and servo level, this feedback Is Implemented via various 
control system schemes designed to operate with a continuous physical world. The RTC, on the 
other hand. Implements a feedback scheme that operates In event space. A more detailed 
description of an event space formulation of the RTC decision making Is given In the next 
section. 

4. Event Control 

The notion of event space and event control la Intimately connected with the time 
horlxon of the subsystem of Interest. It can be argued that reaction time for a subsystem at 
any level of the hierarchy should be sufficient to react to a possible world change that Is 
relevant to the goal commanded to that level. Therefore for any subsystem, its time horlxon 
should be comparable with the time constant of the control processes at that level. While the 
precise determination of appropriate time horizons must necessarily be ad-hoc. It 
nevertheless allows the definition of a hierarchical control scheme, where a higher level 
relies on the lover subsystem for the execution of commands associated with small tlae 
constants. 

A hierarchy in the chain of coaaand Is naturally coupled with a hierarchy In the 
processing of sensory information as well Cl 3. This leads to an Important slaplif ication In 
the overall task execution monitoring activity in the telerobot. That Is, the subsystem at 
the commanding level is relieved froa the mundane supervision of command execution at the 
lover levels, and is instead free to focus on monitoring and anticipating a smaller (and 
finite) number of crucial events. 

Taken together, a "trusted soldier principle" (21 nay be formulated. According to this 
principle the overall process-level decision space can be factored Into two distinct 
components. One component la acted upon by the RTC, while the other la the sole 
responsibility of the sensor and servo subsystems. 


On any level of hierarchy. Event Control requires the napping from the space of sensor 
event sequences: 

{Sj*}. i - 1. .M k K-l. .S n 

to the space of actuation event sequences available for this level: 

'{A i k >. i - 1 ..N r K-l. ,S n 

where *1* is the running index for the sequences and S n represents the number of lover level 
actuation/sensor subsystems. 

While this formalism can be used to describe a traditional servo system with digital 
elements in it. Its main goal Is to create a finite parameterization of actuation and sensor 
events therefore making error recovery through search possible. 

The "trusted soldier* principle leads to a unified command forsat. Commands to a lower 
level specify goals, constraints on the allowable ways to reach this goal, specifications on 
when activity should be stopped, and report formats as well. 
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In our caw th» coaaunlcatlon batvwn tht lavtla of th* want control apact < RTC > and the 
contlnuua apaca (HCH and SAP) la performed by a Primitive which la modelled as follows: 

Prlaltlva T - (P.H.F.R) 

vhara $ 

P * desired trajectory (position x forca apaca) 

H - desired claaa of control law 
F - daalrad raflax pradlcata (atop function) 

R - daalrad avant raportlng function 

Tha coaponanta P and H apaclfy tha phyalcal notion to ba axacutad by tha robot. The 
coaponant F dataralnaa tha taralnatlon of tha apaclflad phyalcal action wla a aanaor 
pradlcata which whan trua triggers tha approprlata raflax action (typically a atop). Tba 
coaponant R dataralnaa tha aanaor pradlcataa that ganarata tha avant raporta to tha RTC. 

Tha apaclf Icatlon of P and H dataralnaa tha daalrad aap froa tha actuation avant apaca to 
tha contlnuua aanaor /actuation apaca. Tha coaponant R apaclflaa tha aap froa tha aanaor 
apaca to tha aanaor avant apaca. Tha coaponant F apaclflaa tha pracoaputad raflax actlaaa 
that auat ba takan by tha HCH and rapraaant tha algabralc (aaaorylaaa) and local coaponanta 
of tha daelalon aap that ara to ba axacutad within tha fast daclalon tlaa conatanta of tha 
HCH aubayataa. We want to emphasize that tha atop function F la to ba daflnad bafora tha 
atart of tha actual coaaand execution. tharafora allowing praplanning for all poaalbla 
outcoaaa of T. 

Evant Control in tha RTC 

Tha lntarfaca batwaan tha RTC and tha lovar laval aubayataa adopts tha "trusted soldier* 
princlpla of oparation. That is. tha lovar laval aubayataa haa tha full raaponalblllty of 
parforalng tha daalrad coaaand T. If T is unsuccaaaful than tha rola of tha RTC la solwly 
that of coordinating and lntarpratlng tha various raporta R (froa all of tha aubayataaa) to 
dataralna tha naxt coaaand T to ba aant to ona of tha lovar laval aubayataaa. 

Glvan this aoda of oparation. conaldar tha actions to ba takan by tha RTC on racaptloa 
of a coaaand froa tha AIP or tha oparator. Tha coaaand aay ba avantually tranalatad according 
to ona of tha four following caaaa: 


1. COMMAND 

2. COMMAND 

3. COMMAND 

4. COMMAND 


~ > (P.M.F.R) 

— > (Pl.Mi.Fx.Ri) 

— ) ( p n*M n .F n .R n ) 

[ (Px .Mx . Fx . Rx) ( P2 * M2 * * * * ( Pflj'Mm . 1 


[(Pll.Mii.Fii.Rii) (Pi2*Mi2.Fi2' R 12) ( p lm'Mi 0 . p im' R lm) 1 

t ( p nl.M n i,F n i ,R n i) (Pn2' M n2 * p n2 * R n2) ••••( P nm • Mnm * p nni* R nnil 1 


Case 1 corraaponda to a slngla poaalbla prlaltlva T (albalt paraaatarlzad ) that can ba aant 
to tha lovar laval aubayataa. An axaapla can ba a coaaand to aova an ara to a glvan joint 
position using glvan joint intarpolatlon. 

Casa 2 corraaponda to tha caaa vhara ona of aany prlaltlvaa T aay ba aant to tha lovar 
laval aubayataa. A coaaand that apaclflaa and position, but not tha apaciflc trajactory to ba 
salactad by tha RTC. can ba uaad as an axaapla hara. 

Caaa 3 corraaponda to tha caaa vhara a aaquanca of prlaltlvaa T la nacaaaary for process- 
laval coaaand axacutlon. and caaa 4 daala with tha situation vhara aora than ona aaquanca la 
avallabla. 

Since only ona coasand can ba aand to a lovar laval at a tlaa. tha aablgulty that exists 
in tha cases 2 and 4 auat ba reaoved. Thera are aavaral poaalbla ways to do It. A "hard 
rule" approach assuaes an axlstanca of an lapllclt agraaaant between tha coaaandlng aubayataa 
and tha RTC on how a single fourplet should ba chosen. For axaapla It aay ba aasuaad that a 
straight line Intarpolatlon should ba always used. In this caaa there ara no real 
differences between cases 1 and 2. On the other hand a "soft rule* would allow tha RTC to 
rank all candidates and start execution fros tha "beat" ona. If it failed, and tha actios 
ware reversible. then another candidate could ba triad. Tha RTC lapleaenta a aixad approach. 
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It ranks possibilities internally based on an RTC internal criteria; but after a choice la 
aade and execution started* the rule beeoees "hard* • and an error in execution would require 
the RTC to report it back to the conserve! lng level without exploiting the rest of the options. 


Coaaanda to the RTC which result in Cases 1 and 2 are called Actions. Cossands 
resulting in Cases 3 and 4 are called Skills. Skills require a sequence of event control 
decision making prior to the Issuance of each lover level subsystes primitive T. In Case 3 
the sequence of primitives is constrained by the one possible mapping indicated earlier, but 
the numerical parameterization of the individual T is determined at run-time by the RTC. On 
the other hand* Case 4 allows the possibility that a different sequence of primitives 
together with their individual parameterization may be selected during the middle of the 
execution of a specific sequence. Since the creation and execution of this sequence is 
dynamically determined by the RTC, the ability to change the parameterization of a primitive 
T, or the actual change of a sequence, gives the RTC the ability to perform "trimming" <l.e. , 
error recovery of subtask command execution). Rote that by virtue of this definition of 
trimming* Actions cannot be trimmed. 

Using an analogy from control theory. the modules that Implement the * forward loop' 
operations of Action selection or a nominal Skill sequence determination are Incorporated 
into a Process Level Planner, and the * feedback loop' that analyzes events and determines the 
need for trimming is Incorporated in Ronltor, Predictor* Evaluator and Trlaaer modules. 
The feedback paradigm corresponding to this event control system is shown in Figure 3. 

It should be noted that two key assumptions govern the functionality and design of these 
modules. It is assumed that a nominal task sequence is known from off line analysis, and 
that in a case of ultimate failure, control can always be surrendered to the Operator. 

For the HCH a particular implementation of command components P, H and R is given by: 

P - a set of via points in Joint/task space, or spline functions. 

H - position mode or compliant aode control. 

R - standard report information returned to the RTC after stop (or reflex actions): 

joint positions* force/torque sensors readings, and the reason for end of execution. 

The choice of the stop functions set F is more complicated. It is clear that F must be 
complete in the sense that some condition must be eventually triggered. The triggering 
conditions aust also be unambiguous, namely only one stop function should be triggered at a 
time. Such a choice of the stop functions makes the number of possible outcomes for each HCH 
command finite. 

The process level planning la based on scripts. This allows the embedding of careful off- 
line analysis of the possible outcomes. In the future an expert system can be implemented to 
make a choice of an appropriate trimming action. Another advantage of scrlpta lies in their 
ability to Incorporate ad-hoc domain specific knowledge for planning of a nominal, feed- 
forward part of a skill. For example an approach position for a grasp can always be 
determined as a set distance away from the grasping point. Another specific feature of the 
current implementation is the existence of a command-object cross table which provides for 
every action, a list of parameters needed for nominal planning of this Action for each known 
object. It can be filled off-line or, if appropriate nodules are available* on-line upon 
demand. The nodular nat ire of this architecture allows system modification and extension in 
an orderly fashion. 

Skill Implementation includes two different phases: planning and execution. First an 

appropriate script employs a generate* test and modify paradigm to make a nominal sequence of 
actions. At the same time the script provides a list of possible trimming actions that can be 
employed to recover from the errors on the intermediate steps. Then the execution is 
performed according to the following loop: 

WHILE Actlon_Staek Not_Empty 
loop: 

Bind Action Parameters; 

Set_up Evaluatlon_context; 

Send Action to the HCH; 

Wait for Report; 

Evaluate Report; 

if SUCCESS then 
null ; 

else 

Determine trimming 

if ANY AVAILABLE then 

put trla_action on Actlon_Stack; 

else 

Report _up< Fail ) ; 
endlf ; 

endlf ; 

endloop; 
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An actual example of a acrlpt to parfora object grasping la ahovn In Figure 4. Numbers 
aftar aach action rafar to tha trlaaing stack associated with tha graap acrlpt. 

3. Iaplaaantatlon 

Tha previous aactlona have daacrlbad aoa a of tha techniques uaad to parfora cartala 
daalrabla alapllf lcatlona. auch aa aapplng a contlnuoua problaa apaca Into a discrete oaa. 
Nov tha atructura and iaplaaantatlon of tha RTC architecture will ba examined. 

Aa background, tha aoftvara Iaplaaantatlon of tha RTC la on an Al VAXatatlon II, uadar 
NlcroVHS. Tha aoftvara la vrlttan prlaarlly In Ada, vlth aoaa port Iona In C and Fortran, 
vhlch ara accaaaad vlth tha Intarfaca pragaa auppllad b/ Ada. Ada vaa our cholca of langaaga 
for tha RTC dallvarabla aoftvara, but va parfora rapid alaulatlon and prototyping of 
algorlthaa in Prolog, Llap, Saalltalk and a locally-vrittan davalopaant anvironaant callad 
Thraad. Our cholca of Ada for tha deliverables vaa aada for aavaral raaaona, tha anat 
obvloua balng that tha RTC vlll avantually hava to coaply vlth tha Spaca Station all-Ada 
aoftvara requirement anyvay. In addition, tha ability to axpraaa concurrent prograaa 
dlractly in tha languaga vlthout raaortlng to oparatlng ayataa lntarfacae vaa alao a 
daalrabla faatura. Our atrongasr reason, hovavar, vaa tha axtraaaly high nodularity of Ada, 
allowing aavaral people to vork on tha a ana aoftvara vlthout dlaaatroua conaaquencea. Thla, 
coupled vlth tha anoraoualy high level of coaplle-tlae error checking In Ada, turns out to 
hava bean a true expectation. Our aoftvara production haa bean only looeely coordinated 
aeong four different people, but It haa axpar lanced no inconpatlblllty problena and haa a 
hiatory of phanosenally lov levela of runtlsa errors. We believe that Ada la a good cholca 
for our dellverablen alnca they ara of precieely tha category of aoftvara Ada vaa daalgnad to 
fit, nanaly dlatributed eabedded real-time ayatena. Wa would make tha consent, hovavar, that 
Ada la not a good cholca of language to prototype In. 

Tha telerobot vlll ba uaad over the couraa of aavaral yeara for aany purpoaes. Including aa 
a testbed m vhlch to lnveatlgate both lov <e. g, feedback control lava) and aedlun (a. g. • 
graap atrateglaa) level robotica algorlthaa, aa vail as for tha targeted danonatration 
scenarios. For thla reason, tha architecture of tha RTC nust hava tha character sore of a 
davalopaant anvironaant rather than of a finished product. Tha ability to reconfigure the 
talerobot to parfora servicing tasks on satellites other than tha one targeted for tasting la 
one of tha obvious requlreaents. It Is our opinion, hovavar, that tha aany eoaplex 
interactions vhlch can taka place during various types of robot operations, even In auch a 
structured situation aa tha servicing of a modular satellite, vould aaka it likely that even 
state-of-the-art robotics algorithss vould ba far too rigid to seat all naada. To Insure 
sufficient flexibility, one must not only provide aa auch aa possible in the vay of current 
robotica algorlthaa, but also provide the ability to completely change hov, when and which 
algorithms are applied. 

Tha ability to do such general restructuring vithln a practical robotics lspleaentatlon has 
very large software consequences since the addition of a particular algorithm to tha alx uaad 
In tha robot might require brand new data representations to accommodate new Information 
relevant to previously unaodeled properties of objects, and the addition of the new algorithm 
could very veil require completely redesigning the strategies used for applying various 
currently-used algorithms. An example might ba tha addition of a new grasp planner module, 
vhlch might require that more data concerning each object to be grasped be installed in the 
data base and that the previously used strategy for stable grasp position determination be 
disabled or modified. 

In order to support such a high degree of flexibility, tha structure of tha RTC 
architecture is that of a framework in vhlch robotics algorithms can be installed, rather 
than a particular six of algorlthaa. The RTC conslats of several modules vlth a very rough 
degree of functionality assigned to each type of module, but the specific Inputs, outputs and 
details of operation which each module performs are controlled by reconf igurable data sets 
supplied to each module. The rough partitioning mentioned above is relatively simple and is 
an ad-hoc solution to the question of hov to slice up tha problem of processing tha commands 
received by the RTC. One important characteristic, namely the ability for the RTC to accept 
commands m parallel, had a major role in selecting tha problem meparatlon. 

The RTC is required to be able to perform more than one coaaand simultaneously, assuming 
that they do not conflict vlth each other. The task of managing access to various shsrable. 
non-shareable and queue-based resources is fairly straightforward in the context of tha 
talerobot, due to the fact that the autonomous system say ba Interrupted by the operator at 
any time. This makes it virtually Impossible to perform any time-scheduling of resources, so 
the simple restriction is made that all commands must ba independent of each other with 
respect to time shifts. This removes tha necessity of performing the many eoaplex sorts of 
resource allocation and activity coordination vhlch vould otherwise ba required between 
commands. This restriction is not as crippling as it seams, however, since there is no rule 
vhlch states that a single command can use only one resource. In fact, if a complex 
cooperative task involving two arms and a vision system is desired, it would simply need to 
ba formulated as a single coaaand so that the coordination operations necessary could take 
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pi ac* vi thin th* fraiatv ork of on* coamnd, without requiring propagation to other activities. 
This independence also greatly ■lspliflM the task of managing rfp oo— to w&nptcttd ar r or 
condition*, nine* all that la nscaaaary 1* a alapia tarml nation of tha command, or of all 
currently executing comm a nd* if tha arror la a global on*. Again, thla alapia bah a vi or la 
not raatrlctlva, alnca any aort of coaplax arror rmaponam can ba produced, if daalrad, a Imply 
by including it aa an axpectad poaalbla off-nominal altuatlon in a com ma nd. Obviously, thara 
la a graat daal of poaalbla interaction between robot action* in the real world, and 
formulating two coamanda in much a manner aa to be truly time-shift invariant might be very 
difficult in any given altuatlon, but thla la not really an la a u a . If the two operation* are 
unconnected, they can alaply be done in mar lam, formulated aa two a a p m rata co ama nd a. If they 
mu at be don* cooperatively, they ran be foraulatad aa one command. The performance of 
dependant operation* in parallel could aid efficiency, but la not a key requirement. 

kith tha above preface, the ad-hoc decompoei 1 1 of the RTC*s procaaalng of coamanda 
follow*. 

Co amend Farming 

• An Incoming command from the higher-level ayatem la flrat converted into m o me 
internal working data structure wear! by the other module*} thla allow* decoupling 
internal operation* from external interface*. 

Script elaboration 

• The command la then examined, and a apeciflc aequence of activitiea la decided upon 
to execute it. together with apeciflc poaalbllltlea for reaponaea to off-nominal 
event* during execution. An example of thla would be a graap command, for which a 
aequence of three atralght-lln* motion* followed by a closure of the gripper was 
decided upon, with the off-nominal behavior of backing up to the atartlng point and 
alaply repeating the entire motion if the graap doea not aucceed on the flrat try. 

Thla aequence la apeciflc in the number and type of motion primitive* to be executed, 
but no detail* are preaent yet aa to which trajectory la to be uaed or what the point 
of graap on the object la to be. 

Action Binding 

• An elemental motlon/aenalng primitive in the aequence 1* then further proceeded, 
in order to determine the precis* numerical parameter* for it* execution. Thla 
primitive la then aent to the appropriate subsystem. which reaponda with on* or more 
report* back to the RTC providing progreaa/reauit information about the command '* 
execution. 

Report Anal/ais 

• The returned report (a) are exaained. and a determination la made whether to 
continue execution, abort or take aome off-nominal corrective action, much aa the 
retry option mentioned above. If the declalon la made to continue, the prevloua step 
la then repeated for the next primitive in the aequence, followed by thi* one, until 
all primitives In the aequence have been executed. 

The overall architecture of the RTC le one In which each of the steps outlined above has 
been assigned to one type of module, with each command thus being processed by one of these 
four types of modules at any one time. Multiple modules execute In parallel and communicate 
by message passing In a very simple way. New instances of each module can be crested as 
needed to process parallel commands since each module le simply a worker which la dispatched 
with a job to do and then returns with a result, with no psrasnsnt memory of Its own mmd 
(with m single exception) no aids-sflscts. All information relevant to the processing and 
execution of the command is contained In the Inputs and outputs to each module. with the 
exception of spatial/geoaetrlc Information about the state of the world, which la contained 
In a global database. This database can be read by any module, but only the module assigned 
to analyze the reporta returned from an executing mubayatea haa the authority to write to it, 
thua updating the information m the database with the data returned by the executing 
subsystem. Thla is the mingle aide-effect present in the execution of any module, which 
removes the possibility of write contention and other such coordination problems. 

Thy RTC le thus mads up of zero or more copies of smeh of the four command -process lag 
modules, together with several additional module* which perform various essential function*: 

• An Interface Server, which serves aa a communication port to the other subsystem* 

In the telerobot. 

• A Decision Unit, which le the central dlepatcher/coordlnator for the RTC. Thla le 
simply a finite state machine, with the four-phase command processing behavior built 
into it. along with the terminate coammnd-on-error and resource- management behaviors. 

In the nominal came. It alaply feed* the Input command Into successive modules, 
propagating one module's output to the next module's input, with minor exceptions. 

In anomalous situations, it alaply takes several straight! orvard steps to insure that 
the executing subsystems and Internal module* working on that command are shut down 
and sends a report of the halt back to the higher levml system commanding the RTC. 
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kcauM of th# fllapllclty of th* Dvclaion Unit Cn alaplt daclaloo tm) v It ntada 
only a vary short parlod of tiaa to asacvta any >?a pona> to on avmt f such aa 
an iBcoalng aaaa agt. For most c a a aa . It alaply atnda out tha locoalog data to t ho 
app ropr lato module. Thla allows wary rapid ra sp o n s a to ex ce ption conditions* such as 
a command from tha telero b ot *s human oparator to stop executing. Tha Daclalon Unit 
will almost cartalnly ba ldla within a law mil lisa c o nd s of receiving tha mammmga. and 
can than pro cass It* and sand out aubsyatam halt prlaltlwas wary rapidly* giving only 

• faw tans of nllllaaconda of daisy batwaan tha oparator * a commanding tha halt and 
tha arm actuation aubsyatam halting arm motion. 

• Command Parsar modulus* aa many as naadad* which conwart tha Input to aa Internal 
working form. It should ba explained that one desirable feature of tha talarobot Is 
to let the oparator Intervene In execution at any level of tha command hierarchy from 
A I ( activity planning Interface) down to detailed are motions (teleoperstlool* so tha 
RTC'e co m m a nd Inputs are also of human -usable fora. Tha c om m and Input from tha 
higher level is thus In the fora of an ascii string with fairly readable content. 
From the RTC viewpoint than, there Is no distinction made between the human operator 
and tha AX planner. 

• Script Elaborator modules* as many aa needed* which decide on the basic script to 
follow for executing the command. In cases where a parameter for aa earlier 
primitive action must be determined by using a precise value for a parameter to an 
action to ba executed later* <e.g.* a precise grasp point may ba naadad to backtrack 
a trajectory to the current position of the arm) then determination of these 
numerical parameters la also performed at this phase* and the -la. ailed parameters are 
Inserted Into the primitive before It la aent to the next phase. This permits the 
use of backtracking aa a planning technique If desired* along with any other 
technique which requires a noncauaal determination of specific parameters for 
actions. 

• Action Binder modules* aa many aa needed* which determine the precise numerical 
values needed for each aubaystea primitive to be sent out for execution. The Action 
Binder would ba Invoked once for each primitive In the script* whether tha primitive 
was a nominal action or a raaponma to soae planned -for off-nominal condition. The 
stepwise Invocation of the Action Binder allows the use of parameters determined at 
one point of the execution of the command during run-time to be used at later points 
In a very simple fashion. Also* as described above* a parameter for an action 
primitive may be already fixed when It arrives In an Action Binder* which will not 
disrupt the normal operation of the module. 

• Analysis Unit modules* as many as needed* which examine the reports returned by the 
executing subsystems* update the global geometric/epatial database accordingly* and 
determine the recommended course of action to be followed* which would be to continue 
normally, abort* or take a specific foreseen corrective action. This recommendation 
is then sent to the Decision Unit* which say choose to follow It* If overall 
execution Is normal* or say Ignore It and choose to terminate the command* If for 
example the AI has sent a halt Instruction to the RTC during the execution of the 
command. The key point concerning the AU modules Is that they operate entirely on 
command-related Information* and do not take Into account such things as global 
anomalies* but operate as a memoryless single-input single-output system. The Input 
is the command context together with the report sent back by the subsystem* and the 
output is the recommendation for action. The Decision Unit worries about 
coordinating any overall behavior which crosses command boundaries. 

• A Database Server module* which simply behaves as a shared-memory area for the 
other modules. Any given location In It can be written to by only one AU at any 
given time* and there are various validity flags to Indicate whether or not other 
sodules should be allowed to read any given piece of Information. 

• In addition to these sodules* which form the basic configuration of the RTC as 
shown in Figure S* there are several others which will not be detailed* but which 
exist because of the requirement that the RTC's geometric database be a ccess ible to 
any subsystem in the telerobot* to act as a central repository. Also* there Is the 
requirement that the AI plmnnar * in order to perform its planning* say need to use 
backtracking methods. To support this* the RTC contains an Identical copy of the 
Command Farmer /Script Elaborator/ Action Binder nodules which are used as a 
hypothesis-testing facility by the AI planner. The planner can simulate as such as 
possible of the execution of s possible RTC command* without actually commanding any 
subsystem activity* so as to determine whether or not a particular line of planning 
will be found to be Infeasible at the numeric level by the RTC. This facility allows 
the fairly ad-hoc division In the planning made between the AI planner and the RTC to 
function robustly In the presence of couplings between the symbolic and numeric 
levels of robot activities. 
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Ths pri mary aaptct of the oodulto which perform command procoMlng la that thty ara 
Implamantad not aa hardwired entitle* which operate on data, but aa data- drive* 
* in te r pr e t er * , * which take aa input a program, aa well aa data. Thle impl em e n tation aXlowa 
the module* to be eaaily reconfigured to uee a new algorithm if at all poaalble. For example, 
adding a new algorithm may abeolutely require a complete reetructurlng of the existing a y at em 
• be ca me* of robotic* conml deration*, but much reetructurlng would not be neceemary elmply 
be cau e e of mof tware dlfflcultie*. Thle feature allow* the addition of many type* of robotic* 
and aleo II technique* to the FTC if deaired einee there ia no reetrlction at all on what 
data la passed between the module*, and only trivial reetrlction* on the order in which they 
are invoked. It la our hope that later veralona of the Script Elaborator will uee Al-type 
reeeoolng technique* to produce the baalc acrlpta for uee during emecutloo, rather than the 
cut -and -paste/ table lookup technique ueed now. Thle vould give the ability to literally 
replan a aeq uence in the event of an error, instead of forcing error r aa p o na aa into a pa c ific 
script*. Aleo, a prediction capability, capable of utilizing expectation information la 
analyzing the tensor data during execution, vould be a desirable feature to add to the 
Analysis Unit. The ability to re mo w the Independency restriction on separate c omma n ds and 
perform coordination between more than one command vould alto be useful in in c r e asing syst em 
efficiency. All of these ldeee haw been explored in a preliminary fashioa, and several 
fairly stralghtforvard implementation alternatives have been found to e e ch of thee, 
indicating that the ability to specify module behavior at run time, rather than Jutt its 
input data, is an extremely powerful feature of the RTC. 

A feature of the RTC is the feet that the memo ryleea, dispatched -worker module format 1* 
ideal for Implementation on a multiprocessor hardware architecture. Modules do not perform 
any significant nonlocal reference* (the exception 1m database access, which can be reduced 
by elmply grouping read requests into bunches, rather than lot* of individual read request*} 
and do not require any communication among themsmlvea during execution. A preliminary study 
indicates that conversion of the RTC to operate on a hypercube multiprocessor vtwld be e very 
straightforward task. 

Another aspect of the RTC'* Internal operation la that it la relatively simple to treat the 
human operator am a subsystem to be commanded by the RTC. This allow several simple but 
effective solutions to the extremely difficult problems of specifying to the autonomous 
systes what the intentions and outcomes of human teleoperation activity are. For example, if 
the operator intervenes Into autonomous execution and picks up an object, there is no vay for 
the autonomous systes to examine are trajectories end gripper force date to determine that 
the object warn grasped at all, or if it warn grasped vhat the position of it in the gripper 
lm. This very simple example show hov difficult is the task of sharing control of as 
autonomous systes with a human operator. One solution, which by necessity imposes a good 
deal of overhead and restriction on the human operator, is to specify intervention activities 
to the telerobot In the same form mm any other command, with the exception that the 
performing subsystem la the operator. This paradigm vould mean that precise numerical 

parameters vould be left out of each step, but the normal aequence of subsystem primitives 
vould be used by the RTC. and likewise the usual sequence of RTC cosmsnds vould be sent out 
by the AI, and the operator vould face the restriction cf performing only that portion of the 
task specified by the primitive. An exploration of this method of structured operator 
Intervention, which Is one candidate for the JPL. telerobot. Is given below. 

If an object could not be reliably grmmp^d by the autonomous system, the operator could 
Instruct the AI planner to 

•grasp Object with Right_arm via Oper*tor_intervention. * 

This vould result in the command trickling down the hierarchy through tbe RTC and m 
subsystem primitive to the operator appearing on the control console. In this example, tbe 
first might be: 

"move Rlght.ara to_neighborhood_of Object*. 

The operator vould perform this. In teleoperation mode, and Indicate that he vaa finished. 
The RTC vould then send out the next primitive: 

•move Right_ara to_gra*p_point_of Object*. 

The operator vould comply, seating the gripper on the object In a satisfactory 
configuration so that when the gripper vam closed. It vould grasp the object firmly, without 
movlno the object. The RTC vould then be able, by performing kinematics and geommtrlc 
computations, to know vhat the grasp point reference frase for the object was and vould. 
therefore, be able to correctly update the data base as to the position of tbe object after 
it has been grasped. The final priaitive vould be sent by the RTC, directing tbe operator to 
close the gripper, and the operator *m response vould confirm that the grasp took place as 
expected, without disturbing tbe spatial relationship mat up. 

This scsnarlo demonstrates the additional overhead Imposed on the operator by the necessity 
of aalntalnlng the autonomous ayatea'a Integrity. It la esssntlal, however, that some form 
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of strong rMtriction b« placed on the operator, . not only to coordinate activity with thw 
autoauMoue syatea, but to p reve nt operator disruption of the autonoaoue eystee duo to buaaa 
oversight. Xt Is likely that there will need to be sore effort put Into protecting tbs 
autonooous sys ten Sr os the operator than Into salting the aotonoaous sy s tem operate 
effectively on Its own. 

Another Issue we bellewe to be of foremost Importance In the telerobot design Is that of 
detection of anomalous conditions In the world. Even In such a highly Identified environme n t 
as satellite servicing. It Is crucial that sensor feedback be employed as often as possible 
so as to prevent any cascade a if errors forward through the execution of e servicing task. It 
Is unlikely that any AX /robotics autonomous system will be able to make allowances for error 
propagation in its activities, within the near future. Such propagation must be eliminated 
If the autonomous syatea Is not to ba c o s e completely confused, with major portions of Its 
world model Invalid, vhlch Is s c ompletely unacceptable aituation owing to the large amount 
of time and effort vhlch vould likely be required In order to restore the world model to a 
co r re ct atate (work weeka). 

The development of such an architecture must nece ssa rily recognize the 11 *i tat lone of 
current science end technology In this nascent area. Early architectures focus oa 
Integrating the process-level autonomous functions into the syatea for simple, independently 
controlled axes. Operator -machine coordination is restricted to simple traded control 
schemes. Later architectures support sore complex physical environments (sore arms, redundant 
area), as veil as sore complicated f unctlonallty much as coordinated arm motion and true 
shared control . Concomitant with this incre a sed complexity end functionality is the 
management of complex computational architectures and the integration of sore sophisticated 
error recovery and planning methods into the systes. At the present tlse, the ETC of the JPL 
Telerobot hms been implemented to contain the following capabilltleet 

• Command Paraer — parses amcll strings fros a simple BNP form into m record data 
structure containing equivalent information. The BNF language in uae has roughly 1M 
terminal aymbola, SO clauaea. 

• Script Elaborator — uaes a almple decision tree to splice portions of eoaaand 
sequences together into a coherent script, together with off-noalnel scripts. Has 
the ability to backtrack from goal point for purposes of trajectory generation. 

• Action Binder — performs generation of jolnt/taak space linearly Interpolated 
trajectories in e plecevlse fashion, using kinematic and load-carrying constraints of 
the manipulator arss, together vith collision detection, to plan small arm motions 
for grasping purposes. 

• Analysis Unit — uses a table-driven decision tree to evaluate the outcoee of an 
execution and makes a re c om m endation. If the recommendation requires corrective 
activities, then the script to perform it Is aiaply looked up In a table, having been 
generated by the Script Elaborator. 


6. Conclusion 

We have deecrlbed a Run-Time Control architecture for the JPL Telerobot and dlscuaewd 
associated iaauea. This work vas performed at the California Institute of Technology, Jet 
Propulsion Laboratory under a contract froa the National Aeronautics and Space Ada Inst ration. 
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1. Abstract 

Tha emer gi ng field of t e ierobotics places new demands on control system 
archkeeture to aflow both autonomous operations and natural human-machine 
interfacing. The feastoiiity of muWprocessor systems performing parallel control 
computations is reafeable. A practical dfetrtxitton of control processors is presented 
and the Issues Involved in the reafization of this architecture are tSscussed. A 
prototype dual axis controker based on the NOVIX computer is described, and resuts 
of ks implementation are discusse d . Application of this type of control system to a 
repkcatsd. redundant manipulator system is also described. 

2. Introduction 


The development ol the field of telerobotics is presently inks infancy. Driven forward by increasing 
ap pk cafb n demands in space, nuclear, underwater, and battlefield actMUes. this new area of technology is 
rapkfly expanding. The advent of more powerful and cost effective computing tech n ology has provided 
solutions to many of the practical problems posed in the development of telerobotic controls. The 
development of a system that can expand wlh future technological pragmas, slow mukiple programmer s to 
sinukaneousty author code, and provide for autonomy as wel as human control is the chalenge that iss 
Immed Uts ty ahead for NASA. To meet these chalenges requires an open archkedure wkh parakel 
processing performance and the abfity to be organized in a logical hierarchy for future expansion. 
Partitioning of such a hierarchy requites that many questions related to performance, expansion path. 
co mmuni cat ions, and software be ans we re d. 


The first step that must be taken to partkion a telerobotic controker is to define the major control 
actMUes and information flow paths/lrates that are associated wkh those activties. A top level Suing of the 
actMUes that must be performed in a telerobotic controker is given: 

1) Servomechanism control 

At the foundation of any telerobotic controker is the subsystem that must doss the control loop 
around the encoder information and the motor drive ampBier to provide stable, responsive 
operation to input drive comm an d s . TradktonaHy, robotic controls have utikzed poskton as the 
Input command and have sensed position and velocity information to accomplish dosed bop 
control of joint location. Uore recently, developments have been made that akow torque control 
to be performed wkh servomechanisms. Different modes of operation may ba required to 
optimize joint performance tor a given task. This lowe s t level of control requires loop closure 
rales from 10 to lOOOHzdependtog on the Hdekty of control that is desired. 

2) Human-machine oommunbalions. 

The key to flextole and mukiputpose teletobotics is the abtty ol the system Mari ace D be made 
transparent to the human operator. Efficient operation of a telerobot in unanticipated 
applications wit require that the human have luk and natural command of al (unctions ol the 
manipulation system. Improvements in graphic displays, master controller*, force-reflection 
capabilities, and viewing methods wil be required to improve telepresence efficiency. 
Computational burdens associated with real-time graphic displays, controller transformations, 
and system dagnostics can result in slowly responding human-machine interfaces unless 
proper dtotributton of computational requirements is accomplished. 
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3) Sensor Integra ti on. 

A primary key to succ es sful autonomous operations is the ability to acquire and decipher 
sensory information from a number of dtfferent sensory systems. Acquisition and fusion of 
vision, facile, force, and scanning information is computationally intensive. Such sensory 
information must be avalabie at rates of 1 to 30 Hz depentfing on the type and quaity of the 
information. Software to extract information from sensory data is being developed by a nuntoer 
of reseaicheis at many institutions to solve spedfic applcation problems. Such cSverse sensory 
development activities wfll conceivably continue requiring a flextoie, but well documented 
sensory communi ca tions interlace. 

4) ActMty/motfon planning. 

Panning is to robotics as the operator is to teleoperatioa For reliable robotic operations to 
occur, the controller must have the abrtty to inteBgentfy act on high level commands and adjust 
actions accotdfog to information returned from the sensory integration system. ThecapabStyto 
mocSfy actions based on the condfflon of the environment is the key to developing a broad 
range of autonomous capacities wfth the teierobotic controller. Equaly important is the abBy 
of the planner to know when the siuation dictates that external human intervention is necessary 
to circumvent a difficuK sfciation. 

5) Internal operational communicationsAcommon memory manager. 

Sequencing between planned activities and actual movement commands must be 
accomplished in a falsafe manner. The internal communications manager assures that the data 
' transfers between the servomechanism controllers and the common memory data base occurs 
in an organized manner to assure that data coisions are minimized. The common memory data 
base serves as a documentafale map of al defined sensory and control data locations. As such, 
it can provide the ability for a number of software developers to independently work on 
subsections of the code without requiring an entirely functional system. For tong term 
evolutionary development that involves multiple software creators, the strict adherance to a 
documented common block of memory win save much time and effort whfle resuMng in a flexfcie 
■ system. 

6) External coordination comm u ni ca tions. 

To allow multiple manipulation elements to interact, it is imperative that an external 
communications handler be developed to sequence and transfer information from one 
manipulator memory common block to another manipulator memory common block. Language 
and communication rates must be delineated in order that the protocol for manipulator to 
manipulator communications may be determined. 

7) Diagnostic handing. 

Overseeing al of the acthrities that occur within the system, there must exist a diagnostic 
harxSng system. This system monitors the basic functionafity of the system components on is 
lowest level and approves the general logic of activities on is highest level. This system 
dtognoses activities from the concrete (temperature, current trips, enables, etc.) to the abstract 
(coilsion between manipulators is emminent, tool not located, you are attempting to enter a 
restricted manipulation area, etc.) and provides the operator with condition and safety 
information that wil protect vafoabie equipment. 

A control system cfagram that shows the interaction between these various processing centers is 
given in Figure 1 . This control dtotribution process is suggested from personal experiences gained from tw 
a p plica ti o n and development of several control systems applied to force-reflecting teleoperation. Theresul 
of implementation of a decentralized structure is a control system that can expand as improved sensory and 
inteEgence technologies become avalabie. The resuting common block approach also allows actwiy 
defintion at an early stage so that muftipJe integrators can work on the development of control system 
software. The activity within a given control center varies depending of the present mode of operation. 
Local inteEgence attempts to replace human intelligence during autonomous activities. 
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Figure 1. Conceptual layout of the principal control centers for teierobotic control. 


3. Statement of the Problem 

The functions described in the previous section can be accomplished in one or several processing 
systems. The essence of the problem is to determine the number of processors, the links between 
processors, the communications structure, and the upgrade development path that wiH provide the desired 
response, expandabtty, and computational performance required for a diverse and demanding group of 
control tasks to be developed over the next decade. Today's answers to these challenges are nether 
straightforward nor deferrable as the technology wfl surely continue to advance. Certain approaches do 
have merit and should be supported in Sght of past experiences and anticipated advances. 

The first architectural question that must be addressed is one of centraized versus dtetributed 
computation. Centrafized processing refers to the use of a single, or small number of centrafly located 
processors to accompfish al of the tasks descrt>ed in Section 1. Wlh distributed processing, a large number 
of less powerful processors perform multiple tasks simuftaneousJy. There are arguments to be made f .r both 
sides. The use of centrafized processors minimizes the communications and timing requirements and the 
synchronization that must occur if mutipie processors must be utilized. Distributed, parallel processing 
s ys t em s provide enormous oomputationaf capabilities within a volumetricaJy smal package. Hardware that 
uses both methods has been developed by the authors. Figure 2 shows the Model M2 control system that 
was developed at the Oak Ridge National Laboratory in 1982-84. It is a unique example of districted (figital 
control for force reflecting manipulator control Utfizing over 30 microprocessors, this controller provides 
closed loop calculations at nearly 100 times per second. Such cfistribution allows online cSagnostics. high 
speed loop closure, variable operation modes, and programmable operations to be accomplished at the 
servocontroQer level The system has performed very reliably for over 2 years of daily operation. The 
repfication of an identical simple software package in each of the joints made the software development very 
efficient The most complex portion of the control development was the sequencing of inter-processor 
comm u ni ca tions. The greatest benefit of this form of control is the very smal communications cable buncJe 
between the master and the slave system. This system could be readily converted to wireless operation if 
d esi re d . The greatest dfeadvantage to this implementation is the susceptabttty of the control components to 
the erwvonment This imitation can be addressed in future systems. 
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Figure 2. An example of decentralized force-reflecting manipulator control. 


Figure 3 shows a general block diagram ola centralized control system. The control requirements of 
typical non-force-reflecting manipulators (fitter greatly from the M2 because the frequency response is much 
lower. As a resul. a single host processor has the eapabiity to sample, calculate, and control afl of the 
functions at a much slower rate (-10 samples per second). If more diagnostic inteligence or force reflecting 
napaharime are to be added to this system, then the computational power must be increased. AddUonal 
processing power does not linearly improve calcula t ional capabffities specifically because of the time required 
to communicate between computation centers. The REMOTEC RM-10A is an example of a 
non-force-reflecting centralized manipulator control system. The loop closure rate only effects the stiffness 
achievable in the servocontrol. but does not resul in any noticable time delay between the master motion 
and the slave. The utilzation of centraSzed control requires handling of a significant number of signal and 
power leads between the master and the slave system. The length at whchthfe type of system w* function 
is also bitted due to lead resistance effects. The centralized control system is not amenable to wireless 
operation without the total redesign of the control electronics. This is not to say that the system has no 
marts. The centrafized control er provides a cost effective, re Sable means of performing Gritted manipulator 
operations. 

The control systems selected tor the Model M2 and the RM-10A manipulators were designed with 
the ultimate performance of the electromechanical system in mind. Force reflecting systems which require 
high throughputs ot information have one level ol control system calculational requirements. 
Non-force-reflecting systems wih Gritted stiffness have a completely different set of control system needs. 
The design tradeoffs result Irom considerations ol prototype cost, replication cost, performance needs, and 
reUbOty considerations. Both of these systems represent an implementation 11 tor is intended function. 

To determ> « the duties of the control system, the next major concern that must be addressed is one 
of automation versus teleoperation. The course that NASA has laid out makes this determination very dear. 
r is a path that begins with teleoperation and evolves toward robotic operations. This means that the control 
system should be capable of both activties. teleoperation and robotics, and the architecture should be so 
devised as to allow expansion paths for future developments leadng to sensory fusion and autonomous 
operations. The deveiopmert of such a control system wS resul in the implementation of true tola robotic 
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Figure 3. Example of a centralized contra Ber for tele robotic control. 


manipulation systems that work equafy wel under human or computer control. The elements of this type of 
control were shown in Figure 1 . The majority of this paper wifl deal with those elements that are common to 
both approaches, the mukimodal distributed servomechanism controBer. By distributing the lowest function 
and highest throughput level of control, the distributed processing approach wiB assure expandabBly of the 
control system to future chaBenges and needs as they develop. The utilization of a common memory system 
allows definlion of the present variable domain and can provide expansion to future variable domains. 

4. Description of the Servomechanism Control System Architecture 

The architecture of the servocont rollers in many ways determines the future expandabitty of the 
manipulator control system. While high level machine decisions are computationaBy intensive, the output 
data that results seldom has as high a bandwidth as the information passage between joints tor mastedstave 
operation. Sensory integration systems may have very high input data rates, but may only have fimftedoutput 
needs. For example, a vision sensory/dedphering system has extremely high input bandwidths, but its 
output may be limited to geometric descriptions of the objects in the frame of view. Simiarfy, an operations 
planner may work on very large data bases, but its output results in high level commands that are not 
communications intensive (scan the past history of successful operations then determine the next step 
needed to accomplish a given task). The following is a description of the bottom-up approach focusing on 
the specific task of self-cfiagnosing servomechanism control capable of multiple modes of operation. 

A prototype dual axis servomechanism controller has been developed with funding from the 
Department of Energy. The controller has l he capability of acquiring data, receiving commands, and 
performing control activities for two servomotors. A detailed hardware description of the device is given in 
Section 5. The purpose of this development is to allow co- location of the controls with the motors to 
minimize cable handling problems, minimize the effects of environmental electrical noise, distribute the 
control complexity to Is most fundamental level, and provide a system that is re&able and easily maintained. 
The resuting archlecture is given in Figure 4. Each of the dual axis controllers shares a common serial 
communications bus and power bus. This yields a system that can be expanded significantly without 
experiencing cable handling problems that so often affect reliability. Using high speed serial 
communications, the serial bandwidth is sufficient to handle between 10 to 20 dual axis controflers. This 
architecture makes ^configurable manipulation readable for special applications targeted at space 
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Figure 4. General aichiedure lor manipulator control using dual axis controflers. 


construction and ground mai nte na nce. Major subsections ot the dual axis control* are the processor, the 
input interface, the output interface, and the two ampBfiers. 

Each dual axis controler has a unique communi ca tions address and can respond independently to 
data requests made from a communications control processor. This processor sequences the data flow 
between dual axis controBers and the common memory area alowing dual ported access to the real time data 
co looted and processed by other processing centers. The resut is a high speed serial communications 
structure for the servomechanism control that is expandable to accomodate additional sensory and planning 
systems. The Kay to the success of this arcMoctura is maintaining high bandwidth for the communications 
between dual axis controllers for teleoperation while alowing provisions for external computer control to 
sequence autonomous movements. The overhead associated with sequencing the dflferent dual axis 
controflers must be kept to a minimum in order that the serial link between controBers hmairtain a high 
thro u ghp u t rate. 

The human-machine interface represents the high level side of the manipulation control system. 
During teleoperation, the human performs the functions of sensory integrator, planner, and instigator of 
activities while the computer performs the role of diagnostician, monitoring the condition ot the system 
hardware. The major activities are completely inverted during autonomous activities as the computer 
senses, plans, and implements motions while the human is placed in a postion ot supervision for the activity. 
This inversion of responsfcOties is accompanied by an inversion of the internal computer communications 
requ i rement s . During tsteopsration. sensory processing systems and task planners are not needed to their 
ful extent, but the comm un i ca tions path between the master and the slave needs to be left unimpaired to 
provide responsive force reflecting operation. Figure 5 shows the differences in the principal 
comnx jn ic a tions flow paths dependfog on the type of operation that is occurring. 

5. Hardware Implementation of the Dual Axis Controller 

A functional block (flagrant of the dual axis controler is shown in Figure 6. The major components 
Indude the processing system (Novix mi croprocessor), the input/output interface, the communications 
interlace, and the ampBier system. These components work together to provide stable, rruKimodal control 
of a dual axfe manipulator element. Operating modes that are ether functional or under development 
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Primary *ctlv* i lwrun ta during autonomous tasks. Primary acthra ala m ants during tstooparatsd tasks. 


Figure 5. Main active elements (or robotic and teleope rated control 


indude: position-position control wth velocity feedforward, position control with integration, velodty control, 
intermediate path generation between communication intervals, torque control, joint initialization, and 
internal diagnostics. In essence, this controller accomplishes at aspects of basic servomechanism operation 
and diagnosis. The Novix computer forms the foundation of this system, and it wl be described in detail. 

The Novix computer represents a new generation of microprocessor hardware. This processor is 
designed in its internal architecture to execute a high level language (Forth) as its ’assembly* language. The 
result is a processing system that is extremely efficient, independent, and compact. Since other popular 
processors (Motorola 68000 series. Intel 8088 series, etc.) operate in assembly languages from which higher 
level languages are constructed, the speed with which the Novix runs Forth programs approaches an order 
of magnitude increase over these other systems. This result occurs even though the present Novix 
configuration operates at a considerably slower dock cycle (4MHz versus >lOMHz). This phiosophy of 
developing a microprocessor engine that executes a language, rather than a machine level instruction set. is 
a concept that will certainly spread to future microprocessor architectures. 

The Novix is available on cel fcraries and can be integrated with other standard ceil devices to allow 
microcontrollers for specific appications to be developed. The potential for intelligent sensors, miniature 
sarvocont rollers, parallel processing/transputing nodes, and powerful man-machine interface drivers is 
quiddy becoming a reality. Some ol the more impressive features o< the Novix microprocessor indude: 

1) Executes 8 mflDcn operations per second of high level codes. 

2) One-cyde local memory access. 

3) One-cyde multiplication and division instrudions. 

4) One-cyde subroutine (word) nesting. 

5) Executes most Forth primatives in a single machine cycle. 
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Figure 6. Dual axis controller functional rfagram. 


The dictionary structure of the Forth language allows the development of libraries of specific 
appfcatbnal words to be shared between progr a mmers on simflar systems. I also encourages top-down, 
bottom-up. or middte-out prog ra mm i ng. This alow* planning of the software functions to be accompfished 'in 
a number of ways afawing future s oftware expansion capabB ti es. Figure 7 shows the hardware reaEzation of 
this system in prototype form. A volumetric reduction of 50% wl be accompished to co-locate the motors 
and oorfroiers within the ioint module. 


XITRecnrc 


Figure 7. Dual axis controller prototype hardware. 
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6. Element for the Advanced TeleRobot (ATR) 


The dub axis controler slows the co-location d controls and mechanisms into an entirely moctolar. 
s#V-contained uni. Tha advantages of this design and construction are numerous. Fust, each module can 
be raplcatsd and uffzed as a shoulder, afcow, or wrist joint. The shared power and communications keeps 
the cable handbig at a minimum. The ki ne ma tic nature ol the element allows reconfiguration ol foe joint 
moMons to alow muUple kinematic construction to be accomp8shed(Le., optimized kinematic anangemems 
lor various task requirernents). Remotely mated mechanical and electrical connections alow quick 
modWc a tion from one kinematic form to another. The basic mechanical element provides dual axis 
manipulation fiat can Bl 30 pounds at 50 inches. Each element weighs less than 17 pounds. For ground 
based appications. a method ol mechanical andfor electrical counterbalancing can be provided. The unit is 
b a ctaHvabto at a pproxi ma tely 20% ol peak bad. Thbalows compliant operation when toe system bunder 
external loads The dual axis manipulator element is shown in Rgura 8- Tha efemait is so versatite Vatican 
be uOfaed as a camera panWK device, a camera posboning device, and arm joint, or a torso poslioner. More 

data! on both themechanfcal and electrical i mpto m ant a ttons can be obtained from Vie reports “Analysis and 
Design Enhancements lor the Advanced Servomaniputator' and Using the NOVIX Computer tor Control ol 
Redundant Teleoperaled and Robotic Manipulators' performed for the Unled Stales Department o I Energy. 



Figure 8. Dual axis manipulator element implementatioa 


7. Summary 

A discussion of servomechanism control techniques tor telerobotic systems has been presented 
and a brief review ol two control approaches was given. The attributes of centralized and decentralized 
control were discussed from a perspective of applied experience. The reasoning behind the dual axis 
controler was developed and the general archleaure reviewed. The actual implementation hardware for the 
dual axis controler has been accompfished and software tor servocontroi has been generated. The dual axis 
manipulator element capable ol redundant, replicated kinematic construction was introduced, and a 
functional prototype was desertoed. These efforts represent a significant effort to move tele robotics ahead 
to meet future chalenges for DOE. NASA, and the DOD. The approaches are novel and taka a large step 
toward a reliable, inexpensive, and adaptable mechanical and el ectrica l manipulation system. Incremental 
improvements were avoided . and completely new ways of accomplishing the objectives of remote 
manipulation were sought. The admirable performance of the prototype controls and mechanisms holds 
promise for future systems implementation. 
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control system for i 
a laboratory environment. 


of the design and ux^ksnentaftioo of a modular, high perfor ma nce. paraDd computer 
Topics of consideration include system architecture, operator interface, and control execution. In 
a telerobotics test control configuration is used to obtain measurements on conanunacations and 


control loop tuning for use in an effective full scale operational system design. The feasibility of the sel e ct ed architectural 
approach has been successfully demonstrated. The modularity of the software and hardware enables ease of tran sp ort for use in 
the operational system, a j«r part inn r*f thi* pyr devoted-to-tb e distributed pardoning of the control algorithms and the 

p e r for ma nce measurements acquired during control system i mp kn a en t ationy^ (/ t * a 


2. Introduction 

Telerobots are manipulators designed to be controlled both directly by a computer and remotely by a human. This combina- 
tion method, called supervisory control, allows the operator to apply his intelligence to the task without having to maintain 
continuous control. Computational requirements for robot control either limit the complexity of the control system, or require 
prohibitively large computers to obtain the required cycle rate. Remote operation and human control requirements create 
additional environmental constraints. The design of a hierarchical, parallel computer system is described. It provides the 
required speed within the prescribed design limits at a minimum weight and size. A prototype system was built md measure- 
ments were made to verify several performance issues. 

3. System Description 

The space-based tele robot for which this computer system was designed (Grumman [ 1]) hes several aspects that drive the 
design of the computer system. 

The first major design requirement is imposed by the robot itself (Fig. 1). There are three arms, each with seven joints and an 
end-effector, a torso, a vision sy stem, and tools and test equipment. All this equipment is under computer control. Each arm 
must operate under control of several different types of control algorithms to provide flexibility to enhance operator productiv- 
ity. Two arms must be capable of coordinated action. The manipulation requirements impose definite compilation requi r ements. 
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A second major design driver results from the control devices used 10 operate in the various control inodes. In order to 
maximize productivity, a variety of control inputs and information displays are needed. Input control devices include a replica 
master. 6DOF stick controllers, voice control, a lightpen. a keyboard, and a touchbezel. Output devices include graphics 
display s, monitors, and force reflection <Fig. 2). Each of these require some data processing. 



Fig. 2 Telerobot Control Station Testbed 


I he third rtuior design driver is that the robot is ; emote f rom the control center. This creates a requirement for a communica- 
tor? link \rs> communication link imposes restrictions and trade-offs on total data bandwidth. 

\ Tina! design driver ;s the set ot different control algorithms required for human control and robotic control. These impose 
requirements tor htferent data .md v *»ntrol paths Different control algorithms have different Ux>p rates K>r stability, wit ha 
•a ’lai a J computations must 'v performed 

4. System ** pec i fit at ion 

I he te:eroh«*t s vauircd To run three different control algorithms Ki lateral Force Reflection iBFR). Resolved Rate (RR>. 
tnd \cf.e <’ vrr \f C I ": : Boih HI R and RR must have human control. \F (' uses computer control * »nl> . BFR 

i is arce .>*n:i;:.*:ii^at.on requirements rial a complete set .»! se:is,.r s 4i ;nais must ne passed both trom the masters to the 
slaves, aru: me 'iaves v 'he masters performance :s ontuined with updates of about H/. \FC has a required 

up fate tor ts command \edor «t about H/ in order to maintain stability u hen the manipulator is m c^tact with elements 

- ‘I N jy t \ irt-nit’e'!* 

Distributed pr-xesMiii: was .oiisjdered to overcome th. computational bottleneck inherent in robot control. First. cad 
aigornhm was ^r. >k.:i nto a m*xk duer.*:n ^v considering which elements were on different levels, which could be run 
.is\ xhror.ousx. ex; which were .omputationallv .merisive. or con id be >.pi *t in?** separate variable types -Fig 4». Then, the 
different a. gonthms were ctMt-tpared to dcvrmme which elements were common Data requirements, computation times, arxj 
output vr each ement were ’abu.ated 























Finally, there re ported dements were cona d ere d in amjmakn with the m fc nnrti oo doer and general system p la cemen t 
requirements to anise at a coherent cocoputef srehi tectn re (Fig. 5). Ptoc eas ot /alg oritom gcrolarity was driven by asaOabfe 
commerc ial p rod a cts and by accuracy requirements. . 

This architecture matches the NASA/NBS standard reference model telerobot control system a r chit e cture (NBS PD far lewis 
one, two, mid part of three. Sesso control and pnramrtrr c al cul a tion take a dvanta ge of parallel proces si n g t eeh alq aes Thsk 
pi inn mg m pwa yacnwnn Mt spur two iucimcocm processes. 



Computer Architecture 


5. ftrfonnance Issues 

Several issues were examined to verily the performance of this parallel co m puti n g system. First, central to the distributed 
control architecture is the correct partitiooment of alogrithms among the parallel processors to properly distribute the comput- 
ing loads. Second, co mimmi ra fio ns bandwidth requirements of algorithm components must be considered to partition the 
algorithms effectively without creating bottlenecks. Third, CPU consumption by algorithm components must be analyzed. 
F inall y, use of operating system and communication system services must also be examined to eliminate unnecessary overhead. 

i Implementation 

The development environment for the digital robot control makes use of Intel microprocessor and microcontroller chip, 
board, and bus technology. The 80286 based System 286/380 is used to run high-level robot path planning code. The system is 
presently used as both a software development and t ar get machine for im ple m e n ting and r unn i ng cootrolsite software. The 
8085-bascd Personal Development System (PDS) is used for developing 8044 microcontroller code, and contains the necessary 
text editor, cross assembler, linkag e editor, and tools for firmware development. An EMV44 eanh i or provides a 

window to the 8044 microcontroller for debug. 

The Distributed Control Module (DCM) package is the set of hierarchical, microcontroller nodes and co m p limen tary com- 
munications software that forms the backbone of the prototype robot control system. It includes the iRCB 44/10 R emote 
Controller Board, the iSBX 344 Multimodule Board, the DCM Controller, and the BITBUS Interconnect. A distributed control 
node may be either m iRCB 44/10 or an iSBX 344 board. The DCM controller is actually an 8044 microcontroller chip with 
built-in SDLC coom rai cntio cs firmware and is part of all boards. The BTTBUS Interconnect is the high-speed serial commum- 
cations link for the network of nodes. 
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The dev el opment configuration was designed specifically to be able to measur e system p erfor ma nce. ll groups six nncrocoo- 
troHers into one distributed network of nodes, tinted together by a high speed serial c ommon i cati oos bos (Fig. 6). The 
co iTmwmic aDoas link inytements a SDLC master slave protocol on chip and perform s all m es s a g e h a n dl in g and error checking 
automatically without help from the CPU. Mastership of the net w ork is daimed by ei th er the Personal Development System 344 
microcontroller node, or the System 286 344 node and is a function of the particular control algorithm currently executing. The 
44/10 microcontroller nodes plug into a common cantfxame which supplies the required power and communications wiring 
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Measuring the system performance is accomplished by attaching an oscilloscope to the pins on one of the dedicated parallel 
I/O ports (Fig. 8). The separate signal lines are toggled *y the development firmware to provide a mechanism for viewing the 
algorithms as they run (Fig. 9). Each bit of the Shit port is dedicated to provide the timing for different components of the 
algorithms running on each processor. 


Measurements were taken in three areas to examine system performance and algorithm partitionment. The three areas 
examined were communications, CPU utilization, and operating system. 
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Serial and parallel message passing techniques differ in the s eq u enc e by which the master node co mmunk ale s with the slave 
nodes. Serial message passing techniques cause the master node to wait for replies after every message sent, which may cause 
other nodes to wait for their tarn if the reply requires some com p utat ion . Parallel message passing techniques aDrav the master 
to have several outstanding m ess a ges, allowing slave nodes mote chance to process in parade! and reduce control 
(Fig. 10). 


The communicatioos bandwidth has significant effect on email timing. With the nodes jum pe r e d for sync hro nou s serial 
co mmun i ca tions on the BTTBUS in t erconnec t, a 2.4 Mbps transmission rate is achieved as comp ar ed with the slower asynchron- 
ous rate (Fig. U). The time savings is readily observed when the master uses serial message passing. 
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Fig. 10 Communi ca tions S ch ama a 
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CPU utilization measurements verify how effectively the algorithms are partitioned. For example, the slave input nodes 
measured a 60% idle time. This reflects a capacity k> handle more of the computational burden of the system. Compautkm 
measurements for different pans of the algorithm equations enable efficient redistribution of portions of the algorithm id fine 
tune the design. For example, the PD loop timing was approximately 0.8 ms. This measurement helped to redistribute the gain 
computation of the servo loop to a more appropriate node to optimize the cycle time of the loop. 


Operating system calls have a significant effect on algorithm tuning. For example, the preloading of a message for transmis- 
sion by a slave node takes approximately 0.4 ms. Internal RAM buffer space (system pool space) must be allocated for a 
message dynamically, via a system call, if it is to be sent by a task on one node to a task on a different node. Since there is a 
limited amount of buffer space (approx. 4 20-byte buffers) available for a message send, buffer space will be unavailable until 
the buffers associated with previous sends are released after successful transmission. A method of reducing the allocation calls 
at the master node is to use the buffer allocated to hold the previously received reply message for the next order message since 
this buffer is automatically allocated by the system. 


Use of the standard RAC communication services may also impose penalties. Passing a message through the RAC services 
automatically causes task context switching. This is important if the node is on the critical path either in terms of calculations or 
buffer allocations . The time penalties can be avoided by taking direct control of the receive buffers. The applications firmware 
must then provide the communication error handling capability. 


8, Cooctusfoas 


Distributed processing of control algorithms is a feasible solution to improve robot control computatio speed. Measurement 
of system elements is important in order to optimize system CPU usage and avoid bottlenecks. Application programs must be 
careful to avoid unnecessary operating system use. 

This architecture, as implemented, covers the National Bureau of Standards Hierarchical Control Specification levels one, 
two. and part of three ^It is independent of path generation and task dt .cription techniques and runs bilateral force reflection as 
easily as resolved rale. Hardware weight and power consumption are very low and are appropriate for space flight hardware. 
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1. ABSTRACT 

Various architectures and their respective software for Hierarchically 
Intelligent Robots are discussed In this paper* They conform to the Principle of 
Increasing Precision with Decreasing Intelligence by following a three-level 
structure. The architecture of the organization and coordination levels Is 
presented here and their algorithms are outlined. 


2 . nrmooucnon 

Intelligent Robots are a special type of Intelligent Machines [51. They may be driven by controls with 
special characteristics described by the methods of Hierarchically Intelligent Control Systems 133. 

Hierarchically Intelligent Control Is based on the Principle of Increasing Intelligence with Decreasliq 
Precision , Intelligent Robots (and thus Intelligent Machines) may be modeled based on the constraints Imposed 
by the Theory of Intelligent Controls. They are considered to be composed of three Interactive levels* or- 
ganization* coordination and execution. They utilize feedback mechanisms from the hardware processes of the 
execution level to the organizer selectively* by aggregation of the Information at every level (Figure 1). 

A three Interactive level probabilistic model has already been defined [4*53 for Intelligent Robots. The 
organization level Is modeled after a Knowledge Based System; It performs general knowledge processing tasks 
with little or no precision. The coordination level Is composed of a specific number of coordinators* each 
performing Its own pre-speclf led functions; It performs specific knowledge processing tasks. The execution 
level Is composed of specific execution devices (hardware) associated with each coordinator. The probabilistic 
model Is obtained by defining the various operations associated with each level of Intelligent Robots la a 
mathematical way and then assigning a probabll Istlc structure to organize the appropriate tasks for execution. 

A simple but complete architectural model that can accommodate fast and reliable operation for the levels 
Individual functions has also been derived [53. Each level has a functional task to perform. The functional 
task of each level Is performed In the best possible way based on accumulated Information and related feedback 
from the lower levels. Upon completion of this functional task* a command Is Issued to the Immediate lo«r 
level* and the functional task of the lower level Is accomplished. With this structure* each level evaluates 
and controls the performance of the immediate lower one. The architectural model associated with top-down 
hierarchical knowledge (information) processing Is suitable for the decision phase of Intelligent Robots. The 
decision phase Involves formulating complete and compatible plans* deciding which one Is the best to execute the 
requested Job and how to execute It. Therefore* it may accept special simple architectures specifically 
designed to Implement the Hierarchically Intelligent Control Algorithms. 

Specific hardware units must be built within each of the higher two levels for the upgrade .phase of 
Intelligent Robots. This phase Involves the upgrade of the individual and accrued costs and probabilities 
associated with a particular plan. Upgrade follows plan execution (coepletlon of the decision phase) and Is 
performed In a bottom-up way: costs and probabilities associated with the lower level are upgraded first fol- 
lowed by the ones with the higher level. 

The architectural model (of the decision and upgrade phases) Is appropriate for both modes of operation of 
an Intelligent Robot* the training mode and the wall-trained mode. The training mode of operation Is defined as 
the mode In which the Intelligent Robot •explores" and "learns" Its capabilities and alternative actions given 
the user(s) requested job(s). While In this mode* the probabilities are significantly modifiable by the learn- 
ing algorithms rewarding certain plans and penalizing others C33. The well-trained mode of operation follows 
the training eode and Is defined as the mode In which the Intelligent Robot is well trained and knows exactly 
the sequence of actions necessary to complete a user requested Job [63. The well-trained mode exists only when 
there are not situations which might include unpredictable events. 

This paper describes both models for the higher two levels of Intel llgcrt Robots. Section three of the 
paper presents the pertinent definitions necessary for the derivation of the models. Sections Four and Five 

2*1 
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present the eodels for the decision-phase of the organization and coordination levels* while section Six ex- 
plains the architectural model for the upgrade phase of Intelligent Robots. Section Sevea presents the 
advantages of the eodels. 

3. i *j» na rrows 

For every Intelligent Robotic Systee define [4*5]; 

1. The set of user coeeands M fixed and finite) with associated probabilities p(c n >* 

n*l»2*..N* sent to the Intelligent Robot via any renote or not channel. 

2. The set of classified coeplled Input coeea n ds IMu^* M fixed and finite) with associated 

probabilities P tu j /c n >* which are the Inputs to the organization level of the systee. 

3. The task doealn of the Intelligent Robot with the set of Independent but not eutually exclusive dis- 
joint sub-sets of non-repet 1 t 1 ve and repetitive prleltlve events E * CE nr *E r ) * 

•• N fixed and finite). 

4. The binary valued randoe variable x f associated with each e t Indicating If e < Is active CXf«I) or 
Inactive (x f «0) given a u^# with corresponding probabilities p(Xj«l/Uj) and p(x^*0/Uj) respectively. 

5. The set of the (2^-1) activities which are groups of prleltlve events concatenated together to define a 
complex task. They are represented by a string of binary randoe variables x j B *( x i ,x 2'* ,iX n>i ; 

■*1*2#..*(2 N -D* which Indicates which e^s are active or Inactive within an activity with a probabil- 
ity 

6. The set of coepatlble ordered activities obtained by ordering the prleltlve events within each activity 
and represented by a string of coepatlble ordered binary randoe variables Y jer' where r denotes the rth 

ordered activity obtained free with a probability 

7. The set of coepatlble augeented ordered activities obtained by Inserting repetitive prleltlve eveats 
within appropriate positions of each Y^and represented by Y j Bp t* s >* "here a s denotes the sth aug- 
mented activity obtained free with a probability p<Y j Br <a s )/Y j Br ) * 

8. The set of eask eatrlces M j Br with associated probabilities pIMj^/Uj) used to obtain the coepatlble 

ordered activities free the activities (X^). 

9. The set of augeented eask eatrlces with associated probabilities pCM^^a^/Y^.) used to 

obtain the coepatlble augeented ordered activities free each Yj^. 

When a user command C ft with a probability p(C R ) Is sent to the Intelligent Robot it Is received and clas- 
sified by a classifier to yield the classified coeeand u^; with a probability plugin)* which is the input to 
the organization level. 

4. MODEL FOR HC ORGANIZATION 'LEVEL 

The organization level performs five sequential functions as shown In Figure 2: machine reasoning* pla- 

ning* .decision making* feedback and long-term memory exchange. The last two are performed during the upgrade 
phase. The organizer formulates complete and compatible plans and decides about the best possible plan to 
execute the user requested Job. This is done by associating with a set of pertinent activities with 

corresponding prooabllltles PCX^/Uj) (reasoning)* and by organizing the activities In such a way (planning) to 
yield coe*>1ete and coepatlble plans: The compatible ordered activities Y ^ associated probabilities are: 

P(Y^ b| /Uj« P (M j BP /u j )tp(x j B /u j K Th * compatible augmented ordered activities Y j Bp 1« s > ace obtained by Inserting 
repetitive primitive events In appropriate positions within each Yj >r and their corresponding probabilities are: 
p(v J«p <a s ,/Y J.r , *P ( * , >r < *s ,/Y J.r , * ,>tY >r /u J > * Ev * ry .ctlvlty and 1i.«wp1ete plan Is rejected WJ. 

Th. most probable collate and cce*>at1ble plan / Is th. final plan that Is transferred to the coordination 
level. 

After the completion of the requested Job upgrade of the probabilities and storad Information fellows as It 
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will be explained In section Six. Figure 3 shows the analytical probabilities Illustration of the complete 
organization level function. 

The architectural Model for the Machine reasoning function Is shown In Figure 4. The Input to the Model Is 

R N 

the (classified) coeplled Input command Uj and the corresponding output* Zj * the set of the (naxlnwa) (2-1) 
pertinent activities with the corresponding addresses which store the activity probabilities PCX^/uj). The 

reasoning bloc* RB contains the C2*-l) strings of binary valued random variables stored at a particular order* 
which represent the activities associated with any compiled Input command. The corresponding addresses which 

D 

store the probabilities related to these activities are transferred from the Memory 0 * a part of the long-terM 
memory of the organizer. The memory D consists of M different memory blocks 0^ D 2 *...»D M . One memory block* 
Dj» Is associated with each compiled Input cowand Uj. Once the compiled Input coamand Uj has been recognized* 
realization of the switch s ^ activates (enables) the memory block Dj. Transfer of the data (addresses) con- 
tained In Dj Is accomplished via the realization of the switch s^. The switches are coupled with each other. 

The contents of the RB are transferred to the right most positions of the PRB (Probabilistic reasoning block) 
while the corresponding addresses which store the pertinent probabilities occupy the left most positions. The 

R R 

Information stored in 0 Is not modifiable by* or during the machine reasoning function. Therefore* D Is 

considered as permanent memory whose values do not change during an Interaction cycle . I.e.* from the time the 
user has requested a Job until Its actual execution. The values of the probability distribution functions 
associated with the set of pertinent activities are upgraded only after the completion of the requested Job 
through a specific hardware unit described In section Six. 

The model for the machine planning function Is more complicated. It Is shown In Figure 5. The Input to 

R P 

the planning model Is Zj . The output from the machine planning model Is* Zj * the set of all complete and 

compatible plans capable to the set of all compatible au^ented ordered strings of primitive events Yj Br (a s ) 

formulated during the machine planning function. All compatible ordered activities are stored In the first 
planning box* PB1. Every compatible augmented ordered activity Is stored In the second planning box* PB2. The 
two compatibility tests (one to obtain compatible ordered activities and one to obtain compatible augmented 
ordered activities) are performed within the boxes CPT1 and CPT2. The specific hardware unit for both com- 
patibility tests Is shown In Figure 5(b). The memory D^* also a part of the 'long-term memory of the organizer* 

Is divided Into four sub-blocks* D T1 * D^, and 0 T4 : contains all Incompatible pairs of the form (non- 

repetltlve primitive* non- repetitive primitive)* 0 T2 of the form (non-repetltlve primitive, repetitive 
primitive)* CX^ of the form (repetitive primitive* non-repetltlve primitive) and D T4 of the form (repetitive 

primitive* repetitive primitive). Each ordered activity (augmented ordered activity) is transferred to the 
register R(R'). At the beginning of the compatibility test the pointer PT (PT') Is at the left most position of 
R (R* J . It transfers every pair of primitives to the two-posftlon register R1 (Rl 1 )* which scans °n (D n' D T 2 * 

and 0 T4 , to check If the stored pair is Incompatible. If yes* the whole ordered activity (augmented ordered 

activity) is rejected. If not* the left most primitive event In Rl (Rl*) Is discarded* the right cost moves one 
position to the left and a new primitive event occupies the empty position. The test continues until the 
pointer has reached the last two primitive events of the activity. Therefore, the number of compatible ac- 
tivities Is significantly reduced. The corresponding addresses of the compatible ordered activities with the 

p 

mask probabilities pCMj^/iij) are transferred from the memory 0 ^ via the realization of the coupled switches s^ 

p 

and s 4 . The switch s^ activates the corresponding memory blcgk (once Zj has been recognized)* while the 

switch s 4 permits the transfer of data. The box PB1 now contains the compatible ordered activities with their 
corresponding addresses containing the probabilities P(Yj Br /Uj). The Insertion of the repetitive primitive 

events Is performed In the box INS while the compatible augmented ordered activities are stored in the second 
planning box* P82. Their corresponding memory addresses with the augmented masks probabilities )/Y. > 

p J ^ J 

are transferred from 0^ . Activation and transfer of data from the appropriate memory block D f4 ^ is ac- 
complished via the realization of the two coupled switches s ? and s $ In a way similar to the ones described 

P P 

before. The Information stored In and 0^ is not modifiable by* or during the machine planning function. 
This Information Is considered permanent within an iteration cycle* too. The output from the machine planning 

p 

function Zj Is the set of all complete and compatible ordered activities that may execute the requested Job. 

The completeness test Is performed within LCMT. This test accepts every meaningful syntactically correct plan 

[5] . 

The model for the machine decision making function Is shown In Figure 6. All compete and compatible plans 

p 

Zj are stored in MDMB (machine decision making box) and checked by pairs to find the most probable one. The 
most probable plan is stored in RR. If during the check* a complete plan with higher probability than the one 
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already stored in RR Is found* ft Is transferred to RR while the already stored one Is discarded. Once the 
check Is over* the contents of RR Indicate the most probable complete and compatible plan to execute the re- 
quested job. 

Every complete and compatible plan Is also stored In a particular part of the long-term memory of the 
organizer* 0 JS « This memory contains every complete and compatible plan related to every compiled Input comm in d 

as shown In Figure 6. The Idea of this particular memory Is very Important: It represents the situation of a 
well-trained Intelligent Robot (under the assumption that no unpredictable events may occur). An Intelligent 
Robot which has reached this mode of operation associates Immediately after the recognition of the compiled 
Input command the most probable of the plans (If more than one available) stored In 0 &$ * without going through 

every single Individual function. 


5. MODEL FOR TIC COORDINATION LEVEL 

The coordination level Is composed of a specific number of coordinators as shown In Figure 7. Its purposes 
to coordinate the Individual tasks* select the appropriate performance requirements for the execution level* 
Identify space limitations and assign penalty functions* optimize the performance of the overall plan and use 
learning for performance Improvement. It issues specific coomonds to the execution level which Is composed of a 
number of execution devices associated with the coordinators at the coordination level. The Interaction between 
the three levels Is represented In terms of on-line (real-time) and off-line feedback Information. The on-line 
feedback information Is communicated from the execution to the coordination level during the execution of the 
requested Job and Is used to evaluate and upgrade the Information stored In the long-term memory of the or- 
ganization level (functions of feedback and long-term memory exchange). 

A block diagram of the coordination level architectural model Is shown in Figure 8. The complete and 

compatible plan Y F stored in RR is transferred to a buffer B. The complete and compatible most probable (final) 
plan, contains actually a sequence of addresses. Each address corresponds to a primitive event (repetitive or 
non- repetitive). When the final plan Is loaded Into the buffer the BEYEMT line Is activated and the pointer PT1 
scans the buffer once from left to right. With the aid of the qualifier QLFR It is known how many times each 
coordinator will be accessed* as well as when It will be accessed. Thus* the association execution devices are 
activated when their coordinator Is accessed via the qualifier QLFR. After the first scanning* the pointer PT1 
returns to the left most position. The BEVENT line Is deactivated and the qualifier Is used to activate another 
coordinator as the pointer moves from left to right. Upon completion of Its operation a BEMT signal Is sent to 
the qualifier and the buffer and the pointer moves one position to the right. When a specific coordinator 
completes Its specific functions and the on-line feedback information has (already) been communicated to It* 
this Information is stored in the short-term memory of the coordination level. It may be used by other coor- 
dinators to complete their functions (during the execution of a requested Job) and/or to calculate the overall 
accrued cost associated with the coordination level* that will be communicated to the organizer after the execu- 
tion of the Job. Thus, the different coordinators do not communicate directly with each other. But each 
coordinator has access to the short-term memory for storage and retrieval of data. 

The coordination level does not have a particular memory equivalent to the D sj m emo ry of the organization 

level. This Is because the workspace environment of the coordination level Is dynamic: Given a final plan * 
the formulation of the actual control problem regarding the way of its execution depends not only on previous 
experience but also on the current conf fguratlon of the workspace environment. For example* previously chosen 
trajectories for the different motions of the manlpulator(s) should be modified by the presence of more or less 
obstacles and/or of additional objects. 

The execution level performs the commends Issued by the coordination level. Each control problem Is 
analyzed In the light of Its special requirements. Therefore* there Is not one general architecture! model that 
can Include every possible operation performed at the execution level. 

But although this Is the case* the execution level consists of a number of devices each associated with a 
specific coordinator and vice versa. Each device Is accessed via a command Issued by Its coordinator. Hence* 
the hierarchical structure Is preserved. 

6. MODELS FOR TIC UPGRADE PHASE 

The architectural model for the upgrade phase of Intelligent Robots Includes specific hardware units t^al 
must be built within each of the higher two levels to account for both types of feedback Information, on-ltiw 
and off-line feedback. 

The off-line feedback mechanism (from the coordination to the organization level) Is activated after th« 
execution of the requested Job and Is used to upgrade the probabl 1 itles distribution functions of the sets ol 
pertinent* ordered and augmented ordered activities associated with a compiled input command. 

The probabll Itles upgrade algorithm Is given by the equation: 

ptt+l/Uj) » p<t/Uj)*£ t+1 ti;-p(t/Uj)3 CD 

I ■ [o 
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where p(t+l/uj) denotes the corresponding probability at Iteration cycle (M)» J the actual cost of execirtlonof 
the Job* J Bi| the minimum cost of execution and 0 a coefficient that obeys Dvofcetsfcy 9 * conditions [33. Each 

upgrade requires one addition* one subtraction and one Multiplication. During this process there are no data 
dependencies between probabilities. A special purpose hardware unit is the one shown In Figure 9. It consists 
of two adders* one aultlpllcator unit and three registers [7]. The probability value Is fetched fro* the 
m e mo ry* passes through the three arithmetic operations and Is retuiaed to the Memory* overwriting the previous 
value. This function Is performed after the execution of the requested Job and calculation of the overall 
accrued cost associated with the coordination level. 

Five such hardware units oust be built In the organisation level to upgrade all pertinent pdfs as shown In 
Figure 10. 

The on-line feedback mechanism (from the execution to the coordination level) Is activated during the 
execution of the requested Job. A block diagram of this mechanism Is shown In Figure 11. Solid lines represent 
the on-line feedback Information from the execution devices (at the execution level) to the different 
coordinators. Individual and accrued costs are calculated within each coordinator and are transferred to the 
short-term memory of the coordination level where the overall accrued cost associated with the coordination 
level Is calculated and communicated back to the organiser after the execution of the Job. The dotted lines 
Illustrate how the Information from the short-term memory Is used by the different coordinators and their execu- 
tion devices for the completion of the Job. 


7. REMARKS 


The models presented here have two major advantages: 

1. They are applicable to any Intelligent Machine operating under the constraints of Hierarchically 
Intelligent Control Systems* and* 

2. The same architecture may be used If one uses a linguistic approach to design Intelligent Machines. 
Based on the models described* one may derive the context-free grammars that satisfy the same perfor- 
mance requirements and criteria. 

The architectural model of the organization level may be also used if one wishes to modify the machine 
reasoning function In the algorithm of the organization level as follows: Given a compiled input command Uj» 

associated with It only a subset of the (2^-1) pertinent activities which contains only those activities with 
corresponding probabilities greater than a prespecified Therefore* the machine planning and decision 

making- functions are limited to the formulation of complete and compatible plans that originate from this subset 

of the (2*-l) possible pertinent activities. This approach requires decision making from the beginning: the 

machine reasoning function because It eliminates every activity with corresponding probability less than 


The disadvantage of the architectural model Is that it does not consider the possibility of unpredictable 
events during plan(s) execution. 
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Abatrti . The problem of co mput er architecture for intelligent robots with partial autonomy is 
addressed. Robot with partial autooomy is considered a degenerated case of a fully autonomous robot. 
Thus, the problem of man-machine coniTnunira tion is formulated, and the are determined 

for generating a language for such a cnmmwni a tiiMi- The dntieurfthg mu«aer Mg — 


Key words: Intelligent Control , Robotics , Mom-machine Operation, Hierarchical Controller, 
Autonomous Robots, Nesting. 


1. Introduction 

Problem of partial autonomy. The problem of partial autooomy is usually considered to be a 
problem of enhancing the capabilities of a teleopeoted pobot Partial autooomy is oteant to compensate 
for the deficiencies of human per ce p ti on and/or performance (DHPP) when the CRT is used as a 
"narrow window" to the world, and no or limited other means of perception are available. After the 
partial autonomy emerges as a necessary p ropert y of an intelligent robot, one can expect to utilize it 
also for simple repetitive jobs, other routine operations, in other words, as a tool of automated 
operation rather then autonomous operation. 

The first term (automated) means independent operation of * machine, following a set of prescribed 
decisions which are submitted by a human operator in advance, In other words, automation presumes 
existence of an explicit, and/or implicit program of operation: all decisions are already made, and all 
possible situations xe taken in account The second term (autonomous) presumes that there are stages 
of planning and decision-making p erf orm ed by the machine with no human involvement In fact, 
autonomy m ea n s that the set of operations to be performed (or "jobs") is not well structured, more 
than one possibility exist, and it is up to the robot to select which one is more beneficial.(This can 
happen not only when the field of view is narrow and/or cluttered by unknown objects. For example, 
making a step upon an unstructured surface by a multilmk redundant leg, gene rates a huge set of 
ill-defined problems. Human operator cannot and need not share his attention among all these 
problems). 

The problem of joint design of man-machine control system was first addressed in [1]. Based upon the 
notion of a limited "bandpass required of a man" the paper suggests that the control system should be 
allowed autonomy in "integration, differentiation, feedforward", and a number of other computation 
procedures in order to allow a roan "to operate as a simple amplifier". The fact that a human operator 
still has to perform a unique job of information integration and decision making, at this stage was 
overlooked. However, very soon the human capabilities to be a universal feedback with unlimited 
"sensor fusion" talents were questioned in [2]. 

Rapid growth of the requirements to productivity and quality factors, brought DHPP mentioned above 
to the attention of researchers in the area of advanced control systems. The variety of tcleopcratcd 
and supervised automated systems described in [3-5] coocentnte upon enhancement of human 
capabilities by allocating some properties of autonomy within the machine control system. The 
state-of-the-art reflected in the literature, implies the sequence of development stages as shown in 
Figure 1. A need in the Interactive Manual-Automatic Control arises in a natural way due to the DHPP. 
According to [5], manual-automatic control allows for some morions under the manual control whereas 
the remaining motions are performed automatically referenced to a variety of sensor data. One of the 
systems created at JPL, features a menu which can be activated by an operator in modifications 
starting with fully manual and ending with fully automated ope ra tion. 

However, the system with partial autonomy can be approached m a different way. Let us imagine that 
the structure of a system is available with full autonomy of operation. How does this structure look 
like? What are the components of this structure that can and cannot be achieved by the technological 
means under consideration? It may hap p en that some elements of the autonomo u s operation can never 
be achieved, or can be achieved in a very remote future. The set of such "unachievable-now“ 
operations can be considered domain of impossibility of the mission (DIM-area).Tben it would 
be reasonable now to make a step back from the ideal image of the autonomous machine, and build a 
system which rely on human participation however only within the limited part: within the DIM-area. 

This approach to the problem of partial autonomy can be illustrated by Figure 2. We can see that the 
supervised automation is viewed as a retreat from die demand for a fully automated system. In the 
same vein, the partially autonomous, or supervised autonomous robot can be considered a trade-off 
between the demand for a fully automated system (which has never been satisfied), and a demand for 
a fully autonomous system (which cannot be satisfied, at least currently). 
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Focus of coaccntyation Problem of antonomous robot operation is often erroneously considered 
a problem of intelligent perception. In foot, even having the problems of perception and knowledge 
organization solved, the problem of motion planning and control of autonomous as well as partially 
autonomous robots lemmas unresolved. This paper a tt emp ts to formulate methods of theoretical 
analysis for partially autonomous pUnnmgtorrol p ro cesses of aclass of intelligent robots equipped 
by a control architecture designed for folly autonomous operation. This control architecture has an 
i mp ort ant distinction which affects supervised op erati on with partial autonomy. Areas of motion 
plMmmg. and motion control are treated separately in non-autooomous systems where such a practice 
is acceptable . Autonomous systems are intrinsically based upon a unified computer system jointly 
emulating planning-control activities as a pvt of a unified recursive computational process.Thcory of 
joint planning-control systems and processes, applicable for robots equipped with Autonomous 
Control Systems (ACS) ascends from the theory of decision making in the framework: of the control 
systems theory. 

ACS were introduced first in [6-9], and the dements of ACS theory are presented there. ACS are to be 
itKtaiiod |n autonomous robots which should operate in unknown environment with limited human 
in volv emen t or with no human involvement at alL ACS are e ye ted to function using human-tike 
procedures of per c e p tion , They have to sophisticated information structure capable of dealing 

with learned knowledge, and communicating with so called, knowledge based control systems 
(KBCSX We make a distinction b e twe e n knowledge, info r m ation, and data. Knowledge is understood 
as a structured information incorporating numerics! data as well as linguistics!, or symbolic 
information, which must be int e rpreted in a definite context It will be demonstrated that the 
principle of nested hierarchies allows for an efficient knowledge organization in KBCS as well 
as for corres po n dingly e fficient processes of knowledge-based perception and control. 

If Joint PlaaningControl Process 

Control solutions to be considered The word control is being used here in its initial meaning 
for decision making activities including planning, navigation, and guidance. This triad is presumed to 
be applicable V least, to systems for control of mechanical motion. It should provide a mapping of the 
a)task,b)knowkdge of the system, and c) knowledge of the environment - into the output 
spccificarions-This triad is an inseparable whole, and each of the elementary pans of this triad can 
operate property only in cooperation and coordination with the others. 

An overview of the recent results in the area of A uton o m ous Control Systems for mobile robots [10], 
suggests that different control solutions show definite resemblance with human teams: they have a 
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hi e rarch y of deorioo-mridnt anitt eren when the qrrtcm h equipped by a single actaatoc [11]. hi 
Hears 3, three examples of control sanctum are compared [12-16). Each rf lime structures consists 
of » namber of goal setting decision nnitt. each tenth to form «n in t ellig e nt module foe automatic 
g engat k m of rootrol st rat egies , policies, and c o cmandi 

However, the most remarkable thing abort all of theie rattans it that m an of them the scope of 

decision maloag is becoming smtlkr, and the l esohrtioo otdocision making ia becoming higher when 

Ihe level of deexnan making is becoming lower. This sequential multiple process of dedsioa nuking 
(replanting, and finding a new control sequence) in the tame situation at differen t resolutions, and 
with different scope of attention, seems to be almost obvions. The major t heore t ical and design 
m hidden in this fact, and thus usually escape from our attention. We would 

as s ume that the underlying theory should be applicable to ACS. 

The major result pronounced within the theory of team control as applicable to robotics, is related to 

the information anrlmr "-rpifrr^ tr> mu rtw Th. this i wfti M twi wnffn. ■'« . 
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special, cont rol oriented knowledge structure. (This subject was addressed in [16], and familiarity 
with the treatment of the problem given there is presumed).Theory of control oriented knowledge 
organization as a part of the ACS, is focused upon development of models of KBCS for motion, 
structures of algorithms, and design of systems for optimum morion of autonomous or 
semiautooomous systems. Theory of ACS employs the fret that the similarities among the existing 
structures of control (mostly, knowledge-based) for autonomous robots reveal a number of inner 
mechanisms of goal oriented dealing with knowledge, as a part of ACS functioning. 

Roughly speaking, any ACS is organized as a team a/ krnman dtciiitn mekrrt which allow for 
using many efficient solutions developed for human teams. Using the theory of team-control an 
efficient p ro ce d ure of computation can be arra n ge d which enables real-time operation of ACS. Thus, 
emphasis is done on solutions which are tractable, and do not lead to NP-hard problems. In the 
knowledge based controllers this problem gets a specific content: information structure should be 
suited a proper knowledge qu an t izatio n. Some steps m this direction were made in [6-9]. 

Hierarchical structures of attention and resolution . In this subsection we will consider 
some of the pe c u liar p rop er ties of hierarchical control sy stems . Hk*?rch>cal decn<rnpc*itk» of systems 
is a well known phenomenon, and is used usually os a method of dealing with their complexities A 
system is bong d eco mposed in parts (parti booed) when the parts contain some ac rive elements of the 
systems subject to control (actuaton). Thes, it is typical to consider multiartuator control systems in a 
form of "tree-hierarchies". Most of the statements in the theory of intelligent control, are formulated 
for the "tre e - hierar chies" of the multi actuator systems, where the hi erar chi cal decentralized techniques 
are recommended: relative independence of the levels is as i m p ort an t, as coor dina tion of the branches 
[17-21]. The hierarchy is being preserved by the fact of resolutiooal as well as attenrional nesting. A 
typical decentralized control system allows for a tree-decomposition which leads to a nested hierarchy 
exemplified in Figure 4,a. 

A degenerated case is turned out to be of substantial importance: when no multiplicity of actuasa: 
exist, and yet, a single actuator needs a nested stem-hierarchy of decision-makers in order to be 
property controlled (e.g. shown in Figure 4,b). Usually, hierarchical systems are convenient as a form 
of organization for large systems which contain a number of goal-seeking decision units 
(subsystems). Thus, the problem usually includes coordinating their actions as to optimize the process 
of goal achievement 

Dedsioa making processes In a nested hierarchical structure. Decision-making process 
which is being done upon the nested set of repr e sen t ation s can be characterized by different frequency, 
accuracy, and combinational power. However, this difference can be stated only if the absolute values 
of these criteria are compared. Indeed, the frequency of decision-making is the lowest at the 
representation "a" (2.HT*to 10" 3 Hz), is higher in representations "V and "c" (10" 3 to 10" 1 Hz), 
and is the highest in rep r es entation "d" (1C 1 to 5Hz). 












Certainly, if the zooming it being continued, the subsequent mag nifying the image win bring 
attention lo the processes of controller compensation, and the frequency of deciiiooHiatang wfll grow 
even more. On the other hod, the accuracy of deriskto-makiiig it the lowest for the representation V: 
the enor of dedtion can be in miles, and then de c r eases down to inches in figure "c\ Certainly , on 
the level of controller compensati on, the re q uir e d accuracy might be even higher. Speaking about 
"combinatorial power", the solutions of represent a t ion "a" can lead to substantial d iffere n c es in the 
overall operation, while the variety of alternatives for the narrow scope of representati on "c“ does not 
seem to be as drastic as in the case of "a". 

However, all these matters assume a totally different consideration. The real difference between 
representations "a" through V is in the resolution of represen tation. Indeed, resolution (or the 
smallest distinguishable foment of die image) in the case "c" is smafl, and in the case “a" is large, 
practically, the values of resolution are d etermi ned by the values of the accuracy allowed). First, we 
notice that the number of "objects" of the world represented mall of the images is the same due to the 
difference of resolution. Secondly, we can easily find that at a great frequency of decision-makmg at 
a level of consideration, the moving robot (s pee d is constant) win pass during the period of time 
bet ween two consecutive decisino-making processes, the same y anrity of resolution units » that the 
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is constant at all levels. After simple tnnsformatioii we receive that 


(resolution at the level) 
for ail levels of consi de ra ti o n. 

Similar consideration brings us to conclusion that "combinatorial po w er" is also the same at all levels 
of consideration; it is just applied to different units of infor ma tion, units that have different resolution. 
This hierarchy of represen t at i on s at different resolution can be easily transformed into a hierarchy of 


Decision making processes of planning-control procedures In the most general form, the 
controller can be rep t t sea l e d as a boa with three inputs, and only one output. Three inputs can be 
specified as follows (see Figure 6*): 

-Task: the goal (Q to be achieved, (and conditions to be satisfied including the parametrical 
constraints, the form of the cost function, its value, or its behavior). 

-Description of the “exosystem", or map of the world including numerous items of 
information to be taken in account during the process of control; map of the world (M) is often 

^^^^brmatioo (I) is the^oniiation set delivered by the sensors in thevery beginning of 
the process of coorrol, and continuing to be delivered during the process of ACS operation. 

The processes within the controller are iTlustrated in Figure 6J>AdJnput j has determined where the 
points SPj and G are situated within the input map M. Input I gives a limited zone Oj^-O^Cin the 





vi c init y of SP^) in which the information set is more reliable, or even exhanstive.Thus, in the overall 
planned tr aj ectory SPj-G'j-Gp a part SPj-G^ can be d eter mined with more reliability than the rest of it. 
In other words, the overall trqectory SPj-G’j-C% from SP to G might be changed in the future if the 
input I will update the map M in the way that the computed (planned) tr aj ectory will become not the 
most desirable alternative. However, SPj-G j part of the plan (bold line) won't be changed: no new 

infbnnatioo is expected. 

Let ns concentrate now on the part of the trajectory SPj-G'j which is assamed to be more known than 
the test of the overall planXabel the point Gj the "new goal", or the "goal for the lower level of the 
nested hierarchy". Notice that instead of the planned curve segment SP-G’j the stripe is known with 
boundaries B +1 . Within this stripe, a new planned motion trajectory can be determined at a higher 
level of resolution pertained to the level (i+1). Again, only part of the trajectory SP-CTi+1 can be 
refined within the sector Oi^i-Oy+i- TO* m tum * us to the more precise part SPj +1 <Tj+i 
which can be subsequently refined at the level (i+2). En e nt ua Dy "new goals" of the adjacent levels are 
becom ing indistinguishable and the the trajectory SP-Gn can be utilized for computing the actual 
control sequence All previously determined trajectories are ptons at different level of refinement 

This consideration includes the following components of the joint planning/control process . 

i F indin g the optimum plan SP-G based upon the map M, am the task formulation (SP, G, 
cost-function, constraints). 

1.1 Computing the alternatives of SP-G. 

\2 Comparison of die available alternatives. 

1 3 Selection of the preferable alternative. 

? U pd ating the map information M in the vicinity of SP by using the sensor information L 

2.1 Recognition of the set L 

22 Comparison between M and L 

2.3 Deciding upon required changes in the cases of: M/I#0, I/M *0, I*M. 

2.4 Creation of the new map in the vicinity of SP with req uir ed deletions and additions. 

3 .Refined planning the path within the updated zone of the map. 

3.1 Determining the subgoal G* (e.g. as a point of intersection of SP-G and the 
boundaries of updated put of the map). 

3.2 Finding the optinum plan SP-G* ( which implies that the whole set of procedures 
mentioned in p.l, should be repeated). 

4. Track the optimum path SP-G*. 

5. Upon arrival to the new point of the selected trajectory (distinguishable from the initial 

point) loop to the position 2. 

The set of procedures 1 recursive through 3 is illustrated in Figure 6»b. Easy to notice, two major 
recursions are presumed within the structureof accepted decision making activities. Firstly, position 
3 2 generates recursion which presumes consecutive refinement of the information within the zone 
which allows for this refinement Secondly, the major loop from 5 to the position 2 presumes 
repetition of the whole set of the procedures after each i ncrem ent of the motion. Thus, the process of 
pooh pUnning can be substituted by a con se c ut ive determin ing of subgoals located closer and closer to 
the current positiooHowever, still remains unclear a) how to d etermin e the trajectory SP*C, and b) 
how to determine the zone in which the refined information should be obtained. These two key 
elements of the overall process will dete rm i n e the number of recursions to be performed for the each 
point of the motion trajectory. 

Substitution of positioning by tracking Theory of tracking control is much better developed 
than the theory of positioning (or control of "pick-and-place" operations). No wonder, we are 
int er e s ted in reduction of positioning problem to the tracking problem. The similarity between our 
control and "tracking the target", is becoming even more noticeable after we realize that the tracking 
trajectory is being constantly rec ompute d in the process of the above mentioned recursion. 

Obviously, our best planned trajectory can be considered as a predicted trajectory. At the each level of 
the system shown in Figure 6, the bold part of the trajectory is a plan per sc, and the rest is just a 
predicted trajectory. However, for the next consecutive lower level the situation is becoming different 
The plan of the upper level is becoming just a planning envelope for the lower level. Within the 
planning envelope assigned by the upper level, an optimum motion trajectory is being refined 
which in turn is considered as a plan for the adjacent level below. The initial part of this trajectory is 
considered me low resolution control to follow (1-st part of the plan) and the rest is becoming 
just a prediction (2-nd part of the plan). 

Since part of SP-G is cosidered as a prediction anyway, and since the co rre sp onding information is to 
be updated in the future, the problem of system’s readiness to incorporate the results of updating 
information, can be discussed. This generates such topics as storing the alternatives of plan, evaluation 
of predictions by some probability measure, and synthesis of contingency plans. 

Commutative diagram for a nested hierarchical controller. Hierarchical decision making 
process allows for using efficiently the full computer capacity which is limited at each level of such 
hi erarc hy (with no branching). In this case, the tree hierarchy of intelligent control coovens into a 
nested h ier a r ch ical controller (NHO. If NHC is acting under the above mentioned constraints 
the process of control allows for decoupling in parts dealing separately with information of different 
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degree of resolution (easily interpreted as certainty, belief, etc.) This means that the degree of 
"fuzzirress" is different at different levels of the hierarchy, and m the nested hierarchy of the 
fuzzy-state automata each automaton of the lower level (automaton-child) h enc l os e d hi 
the corresponding parent-automaton. and serves for clarification of its uncertainties. 

Cons id e ring subsystems as categories C, and the interaction among them as functors F, the 
commutative diagram of the ACS can be shown in Figure 7 (indices mean: s-scasing, p-perceptioa, 
k-knowledge, pc-pianmng/control, a- actuation, w-wori d). Feedbacks are not shown: boxes are 
conn e ct ed by "factors" which cl a r act c rizc the itructaic conservation in a set of mappings of interest 
The bold boriatmtal line sepmes two major different parts of the system: what is below, is a worid of 
real objects, and what is above die bold line, is the world of information processing. All of the 
•'boxes" in figure 7 are fuzzy-state automata. They are easily and adequately described in terms of fe 
automata theory, and provide consistency of the descriptions, c o m pu ter representations, and control 
operation. Then, the search can be done by combining A* and dynamic programming, discretization of 
tb? space is being determined by the level of resolution, and the rules which are formulated within the 



given context. 


nprt LOt*ti 


Nested dynamic programming The principle of optimality of Bellman [22] can be stared as 
follows for stochastic problems: at any time whatever die present information and past decisions, the 
remaining decisions must constitute an optimal policy with regard to the cur r ent information set 
[23].Thesequence of "pUmring-navigating-piktag (or guidance)-actuator control (or execution)" 
appears as a direct result of nested hierarchical search in the structure of information mder 
consideration. Method of Nested Dynamic Programming (NDP) follows from the commutative 
diagrams and analysis given in [6-9]. It stales that the optimum control should be found by 
consecutive top-down and bodom-up procedures, based on the following rules. 


Rule I. NDP should be p e rfo rmed first at the most generalized level of information system 
with complete (available) worid representation. 


Rule 2. NDP is being performed consec uti vely level after level lop down. The subspace of the 
search at each of the consecutive lower levels is constrained by the solution at the preceding apper 
level rec ompu ted to the resolution of the next lower level 

Rule 3. When during the actual motion, due to the new information, the optimum tra j ectory 
determined at a given level must violate the assigned boundaries, this new information should be 
submitted to the upper level (proper generalization must be performed, and the information strecture 
must be updated). This gener a tes a new top-down NDP process. 

Rule 4. When arrival of the new informal m is bounded (e.g. by a "limit of vision"), then the 
re cur s i on of nested process of planning is being done with consecutive process of subgoals crea tion. 

Control Stratified by Resolution into a Joint Multilelel Planning/ Control I recess. 
The nested hierarchy of maps {mj},i« 1,2^.. is the input for plasning/control system. Actually, this 

nested hierarchy is being generated in the process of interaction between the subsystems M and C . 
Let us show how this process of i nt eraction is going 
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Line 1 shows two nested hierarchies: one of them by generalization (of maps {Mj}> and another by 

focus of attention (of maps {nt|}). Hierarchy of sets is obtained from the hierarchy of sets by 
applying NDP-algorithm per level (line 2). In order to do this, a bested hierarchy is added to the 
nested hierarchy . In order to compute a set 4 he results of applying NDP per level, arc enhanced op to 
the meaningful consistent map partition; one of possible algorithms for this is "convex hull" (CH). So 
this system is closed loop level to level top-down and after convergence, the system of controller 
commands is obtained. 

This planning/control algorithm represents actually a hierarchy of blackboards (HUB) in which 
the communication is being rendered among interacting subsystems of the intelligent module.Main 
navigator takes this plan as a suggestion of the milestones, and determines a string of subgoals at the 
way to the first of milestones. This string is sent to the pilot which also has nested stnicturc.The fust 
sub goal in this string might be invisible for the ACS, the visible goal is to be determined by the 
planner within the pilot. Then pilot's navigator can compute accurate enough trajectory of 
motion to be executed. This trajectory is submitted to controller which in turn, has nested 
structure. Its planner determines the next tarcking command, then its navigator determines the 
execution sequence, etc. 

Built-in combinatorial opera tors: team of decision-makers Not only strings or 
combinations of variables (words) as well as combinations of mappings (clauses) will be considered, 
but also combinations of these combinations. Search by scanning, and inclusion when the desirable 
property is met, is one of the straightforward algorithms of combination generation. If the results of 
search are constantly enhancing the input vocabulary, then during the exhaustive search with recursive 
enhancement of the vocabularies, all possible unions, intersections, and complementations of the sets 

are being obtained. The final vocabulary forms a field (F) which means that it contains all set of 
subsets constructed upon the initial set including the results of applying all acceptable combinatorial 
operations, (one of these operations should be later used for decision making process: combinatorial 
search). Using focus of attention we receive a special category of limited fields which do not 
contain all combination of components which is required by a formal definition of field, and yet gives 
more meaningful recommendations without introduction of complicated algebraic structures. 

Using search for alternatives generation. Any combinatorial algorithm is an operator of 
generating plans or solution alternatives for a decision-making process. Consider A* -algorithm 
for the search of minimum path trajectory [29,32.35], or any type of conventional or ’enhanced" 
dynamic programming 1223331,33]. Then a number (value) is assigned to each of the combinations 
generated (preferability, closeness, propensity, cost-effectiveness, etc.) which will enable the 
decision-maker to make his choice under the accepted strategy of decision making. According to 
the existing terminology, the chain of consecutive decisions named policy of decision-making, or 
policy of control 

Since we have received already a hierarchical structure of representation, and N levels are considered 
which are representing the same world with different resolution, the following situation should be 

considered. Given a vocabulary consisting of N elements at the level T (words) wj j, 

wjsjj the set Sj of admissible decisions (or decision N-tupIes ) determined within the boundaries 
determined by the upper level B{ Wj.j } 

S i -W| i ®W2i®**® w Ni m ^ w iK 

2 nd assume the cost-functionals (e.g. presented as distances) 

J i :{w i },B{w i . 1 }-*p i 

then the obtained N-tuple (a string W|W^ w n ) is the i- level hierarchical solution. This string can be 
interpreted as the next subsequent state of the world, or the change leading to the next subsequent state 
of the world (action). In the latter case, the subactions at various levels of the hierarchy, show the 
overall action which leads from one substate to another. Clearly, any decision making process is a 
result of decision making activities of a couple of adjacent levels, which means that the 
planning/control procedure in a result of a recursive computation upon the whole hierarchy of 
representation. Strictly speaking, the decision making at a level require involvement of at least two 
adjacent levels. Thus, the structure of the levels representation is becoming of importance. 
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It is shown in [5-9] that the phenome na of nesting by g en e raliz ation and focusing the attention can be 
characterized best by the operation of inclusion which presumes some procedures unusual for the 
conventional approaches to control systems. Indeed, nesting by generalization is obtained as a result of 
a)grouping the object of the i-th level by the class-generating fea t ure , b)assigning a label to this group, 
c)plaring this laber into the (i-l)-th level as a single object, d)repeating the same for the words of the 
6-lHh level ^nd e)fmally, obtaining a new word at the (i-2>th levcLThen, any class of the (i-l)-th 
level is nested by genermthatioa in its much broader representation at the i-th leveiOn the other 
- hand, after a solution is obtained as a string within the class of the (i-l)-th level, this solution will 
automatically "carve out" a subset within the corresponding much brooder representation at the i-th 
level, of the same (i-l)-th level class in which the solution was de te n nin cdThc result of such "carving 
out" is nested by attention within its own class of the i-th leveL 

Partitioning and nesting of r e p resen tations. Decomposition of the categories of representation 
is done through decomposition of objects and morphism* represented in this category [24]. 
Decomposition imlies dividing objects and relationships in parts, so the components of objects are 
becoming of interest which was not the fact before the process of decomposition. This, in turn, 
implies higher resolution of world representation than before the process of decomposition. If we 
continue with dec o mpo s ition, the further representations are expected to disclose new details about the 
objects and their parts, as well as about relations and their parts. So, in the hierarchy of 
de com po s itions, all of the levels describe the same world with higher and higher level of resolution. 

Changes in time are represented by se q uences of the snapshots in domain of representation 

describing sequences of the world states. Thus, change in the time-scale is intrinsically linked with the 
value of resolution. Certainly, different levels of the hierarchy can be characterized by different 
time-scales. AUthese phenomena are illustrated in Figure & 

Nested Tesselatioa of Knowledge. This system of nested variables and nested mappings is 
described in a Hausdorff nested H^spnce which means that any two different n-dimensicnal points in 
it have at each level of nesting, disjoint neighborhoods. One may assume that our space is obtained as 
a result of natural, and or artificial discretization (tesselatioa, partitioning) from an initial continuous 
space of the isomorphical world description. This space can be considered as a nested stale-space for a 
dynamical system in which the trajectory of morion can be represented This space is considered fora 
given moment of time (snapshot of the world). Then, a s e q uence of snapshots can be obtained The 
conventional state-space is a s up erpo sit ion of all snapshots which describe the process. 

Two types of discretization (natural, and artificial) should be understood as follows. After the problem 
is formulated, it is already based upon some world discretization since the context is represented by a 
semantic network discretizing the world in a "natural" way. However, when the higher level of 
resolution is desired, the natural discretes can appear to be insufficient Then another "artificial" type 
of discretization is applied, and the space is tessdated by (say) a “mechanical", quantitative division of 
larger bodies into the smaller parts. 
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Hus tesselation is dooe in such a manner that the "points of interest" (symbolizing the variables, 
words, or wwfs) are placed to the center of the "tile" of tcss eli tio n at a level (elementary segment, 
grain, discrete, pud, or voxel of the space). Such terms as segm e ntatio n, granularity, discretization, 
and tesselation can be used m ter mi n ent ly. Property of being centere d, bolds if the tile can be derived 
as a system of subsets with unempty intersection. This property should be considered rather in a 
symbolic way: in fret, we cannot discern any another point within the die, this is a m i nim u m grain at 
this level of resolution. 

Resolution of Knowledge In other words, the die of the tes sel atio n determines the resolution 
of knowledge which is defined as a mmimum discrete of information, or minimum wwf which can 

be stated unmistakably. The mimmum centered die will have diameter e and the net of centos emerging 

from this tesselation is named £~net [25lAnalytical details can be found in [6-9). 

The idea of nested tesseladons is coming together with a definition of a single die T( e ) based upon 
nested sphere theorem which can be rephrased as nested die theorem which defines a chain of 

inclusioosfor the hierarchy of mclatiom Thus, e -net is considered to be a set of elementary dies 
satisfying a condition •*» 

T(*0* e 0 ) 3 T(xj, e i > 3 t < x 2* e 2 ) 3 ~ 3 T < x n, c n > 


where xj, xj,--, x„ are the coordinates of the centers of the dies, 

C 1 , £ 2 . — * E n are the radii of the nested dies. 

In the equation of relationships among the dies £| £2 £ Q 

— — ■••• — 
a l °2 

Fairs of adjacent levels. Hierarchies created in this way, satisfy the following principle: at a 
given leveC the results of generalization (daises) serve as primitives for the above 
levet. Then each level of the hierarchy has its own classes and primitives; thus, it has its own 
variables (vocabulary), and the algorithms assigned upon this vocabulary can be adjusted to the 
properties of the objects represented by the vocabulary. This determines the mandatory role of 
combined consideration of two (at least) adjacent levels of the hierarchy. Set of relationships among 
the variables at the i-th level describes the implications which are being utilized for decision* making at 
this particular level. 

On the other hand, the set of relationships at the (i+lMh level describes the relationships among 
classes and thus can be characterized as set of metaimpliculmis (metarules, metaclauses) which are 
governing the process of applying the implications (roles) at the i-th level In the light of this 
consideration, each two adjacent levels can be understood as a complete E. Post production system 
(analog to "general problem solver'* or "knowledge based controller") m which the metarules applied 
to the alphabet of the lower level act as a semi-Thue process of the grammar (261. 

"natural" (qualitative) way (based upon "words" of the context), and the lower level is discretized in an 
"artificial" (quantitative) way (based upon grids, etc.). The process of decision making in this case 
depend on the ability of these two levels to communicate efficiently despite of the different 
representation of the elementary "tiles". The role of human "master" or "supervisor" is critical for 
ope rati on of such adjacent levels since be is the real carrier of the contest knowledge which is required 
for the adequate translation from the language of the (i-l)-th level into the language of i-th level and 
vice versa. 


Reuohttiou of repr esen ta t io n , labeling the class presumes dealing with this class as a primitive 
at the given level of consideration. Moreover, this class (now, also a primitive) is being again clustered 
in classes of the higher level In order to deal with class as with a primitive we have to neglect the 
inner c on te nt of this dass (which might be reflected as new p rop er ti es of the new primitive but with no 
mentioning the initial primitives with their properties). The levels of the hierarchy of representation 
(since they are created by a mechanism of generalization) are dealing with the tame world given with 
different level of richn es s in submitting specific details, level of coarseness (fineness). We will name 
this characteristics of world representation at the level resolution which is a measure of 
distinguishability of the vectors in the sate space. 

Accuracy versus resolution. It is clear that after assigning to the cluster a new class-label (a word 
to be included to the vocabulary), this class-labd is becoming a new primitive with its own prop ert ies, 
the numerical values are assigned to these new prop er tie s , and these numerica l values are chareterued 
by the accuracy depending on the accepted evaluation theory for this particular property (including 
probabilistic approaches, fuzzy set theory, etc.). Clearly, this accuracy evaluation is formally 
independent from the act of neglecting the inner contents of the new primitive. This means that 
accuracy and resolution are formally independent The word "formally" means: in the view of 
procedure to be perfonned.(Consider digitized images (e.g. any "map" of the world) with different 
base discrete of the image. The bigger this discrete is, the less is the resolution of image. However, the 
accuracy of representation at this level can be the same p rovided the accepted value of error upon the 
base discrete). 
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Thus, accuracy presumes the error evaluation in term* of the existence of difference between the world 
and in description within Utt accepted concept of the node L The smaller the vocabulary £. 
is. the more different phenome n a air negl e ct e d . This neglect may entail the increase in error and may 
not However, the smaller Card(£.), or size of the vocabulary is, the higher is the level of 

generalization, and the larger is the radius of the tile in the €-oet Thus, the following relation should 
hold 1 


p > £ 


Card(£.) 

where p determine* (he value of aOowtbfc error (uaccimcy) and Card(£.) determines the value of 
resolution. 

Thus, the information of the world which is required for decision-making, should be explicated and 
prepared for the subsequent decision-malting processes. Map is defined as a subset of the state-space 
which is to be taken in consideration during the process of decisioo- making. Thus, map is a part of 
policy delivered from the leader k> the follower. Map of the upper level contains the maximum subset 
given at the lowest resolution. Maps for the next levels of the nested map structure (top down) are 
obtained using the second apparatus of nesting; focusing of attention. Thus, only one part of the map 
is being activated; which is required for the current decision- nuking. 

The system of maps has dual nesting: 

firstly, we have a subsystem with nesting by generalization 

see a 

Mj o M 2 d ... o Mj_| z> Mj 

which we name "CARTOGRAPHER", and assign to this subsystem main******* of die information 
rather than active participation in the process of decision-making. 

and secondly, we have nesting by focusing attention superimposed upon the nested hierarchy by 
generalization , and this information extracted from the cart ogr ap her (m^a-l^Z-. is delivered to the 
"C system ( Figure 7 ) in a form of "maps for decision making". 


h | f« I I b k b 

(m x € M ,P(m 2 € M 2 ) mj.jc (nte M,) 

£* fa fa fa 


All fa-predicates are performed on the basis of NDP-algorithm results and thus belong to the 
subsystem of control, all g-predicates are prerogative of the system of map maintenance. Later we will 
show how these predica tes are built in the algorithms of planning/control and map maintenance. 

The problem of map maintenance is of scientific and practical importance. The upper level map 
f planner s map") should be maintained for a long time due to the largest scope, and the "slow 
rhythm" of this level. Changes in the upper level map are not frequent Maps of the subsequent levels 
are to be regularly updated with increasing frequency but decreasing volume. At the level of "pilot s’* 
map, most of the information might be considered of little importance in long terms, and the map t& an 
"ephemeral" one. The lowest level map ("controller’s map'*) may or may not need even an ephemeral 
map maintained as a part of the nested hierarchy. (Actually, fhxn our first experience of dealing with 
ACS we found that intelligent module cannot afford maintenance of the pilot map , Le. of the lowest 
level of world representation), and therefore all processes related to the real time operation have 
ephemeral structure indeed, with a number of logical filters determining whether this ephemeral 
information contains anything k> be included in the maps of the upper level 

The rules of assigning for the information to be retained in details, retained in generalization, or 
droppe d , cannot be formulated consistently for all possible situations. How tong the information will 
be needed can probably be d e t ermin ed only by a master. 

Generalization: belonging to a meaningful class. So. knowledge in a context can be 

represented as an £ -net at a definite resolution, and as a system of nested £ -nets with different scale 
where scale is defined as a ratio betwe en resolution at a level and resolution at the lowest level 
dearly, each of the larger tiles at the upper level is a "generalization" for the set of smaller tiles at 
the lower level of the hierarchy. Selection of one of the tiles at the upper level (focusing anenoon ) 
entails selection of a definite subset of smaller tiles at the lower leveL 
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We would like to urtu die fact that the mctasioo XdXdx show* in thit hierarchy of the tile 
e mbed din p has mote i m port an t and broadmeaning than just 'scaling of the sixe*. The inclusion 

predicate j has a meanit^ of “belonging to a class* (since it is s ss nmrdih s t the objects are unified into 

a set by some class generating property, or featare). One can talk about state space, space of weighed 
prooerties, and so on, and the notion of “belonging to a class of come spatial neighborhood" is 
becoming closer to a meaning of "generalization" as if it is understood in the discipline of logic. Then 
dicretizacion of the state space roil contribute simultaneously to a) minimum required interval of 
consideration, b) hierarchy of dames of belonging to a meaaigM neighborhood. 

From the theory of pattern analysis and image recognition we know that selection of the meaningful 
neighborhood is a difficult issue, and it c an not be addressed exhaustively unless a human operator, or 
master is involved in the procedure of the feature assignment. 

Look-up Tables. The prespecified lists of mappings upon a vocabulary, can be organized in 
look-up tables (LUTs) for convenience. For example, a control “u" can be selected from this 
prespecified set if LUT contains clauses (e.g. “If S *C then o' where S-is a slate description. 
C-strategy selected, or cost-functional agreed upon). Minimum look-up table can contain only one 
mapping. 


Sets of variables to be considered simultaneously are named vectors, or s tr in g s . A single LUT, or 
a set of LUTs axe named grammar (operators, transformations) if they can be utilized to 
generate output strings from input strings. Input and output vocabularies tog et h er with the grammar 
are named language (over a finate alphabet), or a utom a t on (free monoid gener at ed by the alphabet ). 
Clearly, vectors or strings are rt a t e men ts in die above mentioned language (built upon words of this 
language). The rules of combining variables into vectors, or rings, or strings or vectors, etc. are 
being det ermin ed by LUTs . 


Special role of attention When a part X .is detached from the "whole" X (which may happen 
to be category, object, and/or morphism) and the relation of inclusion holds 


x=x p 

this separation of Xp from X we will name focusing attention upon Xp . Sampling is one 

of the common methods of focusing atteation.Usually we focus the attention when the subset of 
attention can be considered important, or is typical for the whole scl The latter case links focusing 
attention with mechanism of generalization. 

Special role of attention is determined by its close links with the overall goal of operation. Attention 
always presume sacrifice of some information in favor of the information set which is considered to be 
more important Thus, attention almost always rely on a set of pref ere n ces that can be ultimately 
formulated only by a master. 

Generalization oriented tend at ion We would like to stress the link between the generation of 
entries for LUT as an approximation in the space of consideration, and the problem of 
generalization oriented tesselation (discretization of the space). At the level of discretization 
determined by the context, i.e. determined by the words of the semantic network, tte results of 
tesselation are always determined by the vocabulary of the muter. 

The subsequent problems of "digitization” aid "quantization", or quantitative tesselation at the lower 
level are being solved currently by introduction of the "shah function”, and ocher mechanisms of 
sampling which has been proven to be applicable in decades of development of communication and 
control systems. Some difficulties, such as aliasing, and similar, lead to algorithmic aggravations at 
the output In the context of this paper, discretization of the space does not allow for the problem of 
aliasing because there is no information between the adjacent tiles of tesselation (digitization, 
quantization), and the information about the tile properties (values for them) is the set of average 
average values over the tile. Clearly, the term "avenge” in this context, acquires a somewhat unusual 
meaning of “ class generating property” .However, in most of cases the”automatic” tesselation of 
the space creates discrepancies when the corre s pondence betwe e n the two adjacent levels is to be 
determined. These discrep an cies might be eliminated by the interference of the master. 

Complexity of tesselation: C-entropy .Complexity of LUT is being evaluated by computing the 

€ -entropy (by A.N. Kolmogorov [251 ) for a space where the corresponding e-net has been 
constructed 


H^-logN^S) 

where S-is a space represented by LUT, with e-net assigned. 

Ng-nutnbcr of elements (nodes) in e-net. or tiles in the tessclatioa 

For the system with combinatorial search as a tool for planning, the category of consideration can be 
repre sen ted as a power set, then ‘nvialiy 


261 


HgfSJ-NrfS) 

Variou theori es of planning are b as ed apon the idea of sp a ce tesselarioo (tj, confi gurarin space, 
esc.). A multphctiy of diflGc^eot toaepet of space tesadatioa is ccesacd, and numerous techniques of 
evaluating the thstaace, sad the c o m p fcxity are known from de literature. It isemential lo undentand 

that all of these techniques aacend lo the theory and fo rmalisms of 6-octs. In oar case, tninimiz a ri oa of 

the e-entropy allows for detti mining an o p tim a ip n a mber of the levels of the hierarchy of a control 
system. 

Acoaracy of rtpiwftohra. To all words and ail edges with the co rr es pon di ng strength of the 
bond, the values of a cc uracy are assigned Accuracy is understood as the value of the relative error 
of the quantitative cha ract erist ics assigned to the node or the edge of the graph. This evaluation can be 
baaed on estimation of probabilities, z o nes of un ce r tainty, anchor in any another way. The approach to 
the world representation in this paper does not depend oa the particular way of dealing with 
uncertain ty evalnatkmThe value of uncertainty is assigned tote numerical I values associated with the 
properties of the tile of representation which is con sidered as a uunimum unit at a level. 

Evalaartou of the tee of a te This is similar so assigning a definite e-oet H ow e ver , this also 
suggests that the nunimam value of the radius of the tile, cu be time d e pe n de n t In order to avoid this 
predicament in the following treatment , are will try not to get involved with the particular stochastic 

and dynamical micr o s tra c tate of the C-tile at the level of the ^pres entation hierarchy. We will define 
the radius of the tile by the width of the fictitious uniform distribution which has the same value of the 

conditional entropy as the real distribution of the variable x. Then, using the value of conditional 
entropy 

H- - J pj(x) la pj(x) dx 


for a Gaussian the diameter of a tile is determined 

A- o (*e/2) 5 

Referring to the definition of C-net, we would suggest that this value of A is the minimum siae of the 
tile radius to he assigned if the net has the same law of dtsaibuboc in all of its nodes. For systems 
with stationary random processes, evaluation will be sufficient with no invo l v emen t in analysis of 
stochastic processes anymore. Certainly, this depend on the nature of what is considered to be s 
random c o mp on en t of the variable. 

Nevertheless m real situations, it is very difficult to co n s tr u ct a level with the same law of di s tribution 
in all of its nodes. The vole of master here, is either to d eter m ine the "average" law of distribution, or 

to allow for a net with different sizes of tiles. Based upon condition for A, the problem of different 
laws of state-space tessclation known from the theory of morion planning (e.g. potyhedra, etc.) is 
virtually diminaaed. The g eom et ry of a single tile become s irrele v ant , and yet the space is fully covered 
by these "fuzzy" tiles. 

IIL Hybrid Master-Dependent Representation 

Knowledge Bases: Semantic Networks with context oriented interpretation 
Knowledge Bases (KB) are considered to be a collection of well-formed formulas (wffs) for 
man-machine as well as for autonomous decision-support sy s t e m s of different kinds. So, it is clear 
that knowledge consists of linguistics discretes (entities) which are later referre d to as units of 
knowledge. When we are talking about knowledge, the represented knowledge is actually 
meant, (Le. we are not interested here in discussing the name of the reality represented within the 
system of world representation). Nesting is being introduced often in a semantic way , and reflects 
semantic discretization of resolution in representation reflecting human experience (which sometimes is 
irrelevamlThtts. in a nested system, a nested hierarchy of semantic networks can be formulated. This 
semantic nesting can be int er pret e d within a context only under human supervision since the subtleties 
of the context knowledge must be properly transformed into statements of existence or absence of 
relationships among the entities. 

Thus, under masters supervision we receive, two types of aested hierarchies for declarative wfTs: 
existential nesting (nested statement of existence) including nested statements of objects and relations 
among them, and transitive nesting (nested statement of change). Both of these nested units of 
knowledge are presented in implicative form (nested clauses). It seems reasonable to describe the 
set- theoretical on one hand, and on the ocher hand, vector-analytical m anners of describing situations 
(which do not differ in essence). This another form of representation in turn, can be divided m two 
different kinds of representation: continuous (using differential equations) and discrete (using 
difference equations and/or finite state machines (see figure 9). In both cases, the property of nesting 
is applied. 

If a hierarchy is built using the existing semantic network, them each (i- 1 Hh level of the hierarchy will 
contain generalizations of the words of the i-th level This means that the solutions found within the 
(*-l)-th level will automatically "carve out" subsets withia the classes of i-th levels in which the 
process of c om b in atorial search can be repeated, and a new solution can be found with higher level of 
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However, the i-th level should 


the (i-I)-th nmiy on the bo undari e s of the 


Postulate of wlHpli represen tation . So. the same world can be represented within the same 
levels of resolution in a different way. Representation is defined ■ a s tru c t ure (e.g. algebraic, or 
infor mation st r ucture ) which is h o m o mo rphic to the world, to the structure of reality (or a domain of 
reality). Representation con ti s tt of both numerical as we* as descriptive informati on about the objects 
and systems, and is luine d to be ob t ai ne d hunt prior experien ce, and or derived t h eor eti cally (based 
up on multiplicity of existing and potrible toote of logical reference). 

The p h e no menon of multiple world rep t wj u ari on presumes that b o mont o r phisms exist between 
various rtpic re ntarions.Cuncntly , various bodies and t echniq u es of world representation are used in 
the p racti ce of control, based upon di fferent el e men t s and rules of information org an i z a tio n In this 
paper, knowledge bases as a technique of world representation, are being focused upon. Knowledge 
Bares as a model of information system, is a system of world repr esentati on which is considered to be 
homomorphic to other systems of world rep r esenta tion such as systems of difference, or 
differential, or integral equations, and others (see Figure 9). 

Nondetermi nistic nested refe r enci ng. Accuracy and reso lut ion can mistakenly be understood as 
noodeterministie p rop erti es of re pr esen tation. We win repar aie the ch a racte ris tics of accuracy and 
r e s olu t ion from the nutidc tcnnmistic information Let the vector of observations be 


Xo-Xod + S 


where deterministic model after re co g niti on. 

£ -stochastic co m ponent after observation. 

Definition. A component of observation it named s to cha sti c component if it is 

a) not identified (yet) with any of models stored in the knowledge base, 

b) substantially larger than meas u r e of ac cu racy and resolution ( I $ >p), 

c) presumed to potentially affect the results of decisaommakng. 

Our approach differs from the classical only in a sense of (1 1) which implies two recommendations. 

Rectmrntwiarioml. Control problem is to be solved as a deterministic problems dealing with models 
Xpd and estimating likelihood or plausibility of the decision, or the policy by measure of uncertainty 

Imbed with the set of {^}- 

Recommendatiom 2.. Learning will be understood as a tool for extracting new recognizable models 
from { ^ } rather than for updating knowledge of probabilisti c c har acteristics of the set { ^ } . 

The following structure of dealing with unrecognized (unaodelled) information is implied by these 
two recommendations. Initial decomposition should be be repeated recursively for the nested structure 







of information. At each level the component is being de co mposed in two parts: which can be 
lecofn^edandiiidiaMiathedetennuasticpaitof the next level, and which at the next level remains 

still u nre c ogniz able (with E[ ^ 1-0) 

x 0i- x odi + $r.i+I + *i 


where deterministic model after recognition at the level i, 

of the stochastic com p on en t * thei-th level which will be 

recognized after observation at the (i+l)-th leveLCtrend"). 

-part of tbe s t och a stic co m po n en t at the i-th level which remains unr eco g niz e d . 


This recursive analysis of the stochastkhrfbnnatkm can be iUostraied as follows 

Xoi« x odi + ^rj+l + x 0,n-l“ x od,n-l ♦ ^r,n + *»n-l 


^j+l” x o<U+l + tr,i+2 + ^i+1 ^.n" x od,n + 


and n is the level where the rec u rsion slops (no consequtive levels are expected to be built). 

This deco mp osition of information (which is possible within the nested hierarchical structure) allows 
for multiple re fer e nce system. Tbe key motivation for the multiple referencing is simplification of 
information representation per leveL Multiple ref eren cing is indirectly presen ted in the requirement that 

E[ ^ ]*0. This means that the origin is placed in a point is the state space as to provide E[ 5j ]*0. 

Then, the rest of the information allocated for decision-making at this level is re f ere nced to these 
origin. 

Another important implication of mnltlpfc* referencing in dealing with nondeterministic 
information, is related to the topic of learning. As mentioned above, the system is supposed to deal 
with partially, or comp let el y unknown world. Thus, Searakg is presumed Any learned information is 
being identified with memory models (patterns) which determine the initial referencing. The residual 
information is supposed to be collected, and later it is expected to gen er at e a new pattern upon the 
multiplicity of realizations. If generation of a new pattern seems to be impossible (no regularities are 
discovered), the change in die initial referencing might be undertaken. This philosophy of dealing with 
new information is to be utilized for procedures of map apdating 

We can sec also within the body of this problem of nested referencing, a direct link among the 
quantitative characteristics of the system and its linguistics! description, and the components of this 
description. At tins time, however, we will restrain from further statements w* these inks since we do 
not have enough factual observations. 

Recognition and Identi ficati on. All of tbe above statements are based upon Definition containing 
undefined word “identified", hi further considerations, the word "recognition’* is u s ed Some hit of 
patterns is presumed winch should be used for compari son and identification. Recognition is also 
loosely defined but some freedom seems to be allowed for possible enhancement of the list of existing 
patterns. Most of these mailers deserve a special and detailed treatment, they are not discussed m this 
paper. However our a ttitu d e toward this domain should be at least illustrated by an 

An example of interpreting the situation with identification, recognition, and learning the new preterm 
of the entities of the world, is shown in Figure 10. If A is a robot location, and B is the goal, then 
min i m u m time path can be sought in one of the following areas: 1, U, and QL It can be demonstrated 
geometrically, that the minimum distance path can be found in the area EL Howeve r, since each turn 
within this "slalom-type” area is linked with the losses of time (speed at the turning point should be 
reduced to avoid s k i dd ing ), then the m i n i mum time trajectory seems to be within one of the areas I or 


It seems, that even before we start to compute all and possible trajectories within these areas, they can 
be analyzed as some entities with common properties. Indeed, area I can be described as an area 
which has “obstacle 1“ on the right of the moving robot all over the motion, area Hi has "obstacle 2" 
on the left of the moving robot, and area 0 has obstacles on both sides. Areas I and IB do not seem to 
be different when robot is moving far from the obstacle, but they differ substantially if we try to make 
the length of the trajectory as small as possible: the trajectory in the are I will have 5 linear segments 
and four turns while the trajectory III wtO have only two linear segm e nts and only rere turn. 
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Farther analysis shows that the cost-functions have different form for all three areas. Moreover, 
within one particular area cost-function can be co nsi dere d moootooo c in the sense of [32). These kind 
of areas are named "topowaya* (33), and they can be considered entities for minimum time problem 
solving at the higher level of plannmg/control system. Rales can be formulated in which the 
co ndition part wffl describe the relative obstacle location while the co ns eq u e nt part will be related 
lo the most general description of the topoway recommended for farther consideration. 

Cleariy, rules of this level of generality have id be recognized by a master, however they can be 
utilized with no master's direct p arti cipa tio n . 

On the eqai valence between the difference equation model, and the production rules. 
Difference equations can be easily transformed into produ cti o n roles. Indeed, instead of the form 

xflt+1)- A<k) x<k) + B(k) o(k) 

where k-is the number of the stage of r ec ursi o n , 

A(k) and B(k) are the matrices of system parameters,' 
x-isa state, 
u-is a control. 

Instead of having this information in a form of an equation one can state 

IF {x(k) A A(k) A B(k) A ii(k)} -> THEN {x(k+l» 

which actually describes the causality exposed by a particular system : "if present state is given, and 
the parameters are known, and some definite control n applied, then the state will change as follows". 
In most of the real systems causalities obtained exper&aentally and/or from the other models of 
representation can be inverted in a form of rukt-prescriptions [28] 

IF{x(k) A x(k+l) A A(k) A B(k)} -> THEN (u(k)} 

which can be in terpr e ted as follows: " if present state is given, and the the following state is required, 
while the paiu nct en of the system are known, then apply this controT. 

In some cases even the form of trajectory can be utilized for rules form u l ation. Indeed, the form 

k-1 

x(k) = F(k,0) *(0) + ^F(k, 1+1) BO) u(l). and 
1=0 

F(k+lJ+l) = A(k) F(k, 1+1) 


implies the following ways of subse q uent planning/control activities: 

1) solution trajectory oo-line computation, submitting to the execution controller, and storing 
in memory for the subsequent use in a similar situation, 

2) retrieving the previously stored solution for the same or similar conditions, obtained from 
off-line training, or horn on-line experi ence. 

Applying the production rules at different resolution levels. The above consideration can 
be expanded if the prior behavior is taken in consideration in order to estimate how the reality differs 
from what was expected and provide correctives for the control signal, or "tuning* for the matrices A 
and B (using available techniques of extrapolation). 

Thus, the following sets of infonnation can be considered initially known: 

S -states : (x(0), x(-l), x(-2),~.} 

P -parameter, : {A(0), B(0), A(-l), B(-l), _} 

G-goal states : (x(k), x(k-l 

The general form of a control-generating rule can be represented as follows 

IF [P A S A G]* -> THEN DO U 

where U -is a string {u(l)» u(2),*~, u(k-l), U(k)}» 

•-means that the is Pareto- optimal 

Solution can be found by one of the selected search procedures. There is a set of optimum solutions, 
the cost difference between them can be found only at higher lesolutionNDP is done a the following 
sequence of procedures: 

SUB S T ITU T E THE "OUT-OF-REACH" GOAL BY AN ACHIEVABLE GOAL 
FIND A CONTROL-GENERATING RULE 

IF THE CONTROL-GENERATING RULE IS NOT FOUND, APPLY SEARCH 
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SUBMIT THE SOLUTION TO THE NEXT LOWER LEVEL 
IF THE SOLUTION IS NOT FOUND, REPORT THE PREDICAMENT TO 
THE NEXT 

UPPER LEVEL 


Thi* approach is very general *nd produces the trajectories of morion in iD ca t e*: when the general 
ink exist and is known to the plxmnn (/control system, or when it does exist but is unknown. All 
a) tori duns of path-search in 2D binary world start with a statement IF THERE IS NO 
OBSTACLE BETWEEN THE ROBOT AND THE GOAL, GO DIRECTLY TO THE 
GOAL, ELSE DO THE SEARCH [29 J. In fact, this ink mast be proven by g eom etrical methods 
(Le. at the higher level) or foang by a search after disexetixation and petting the problem to the lower 
kveLlhis is the way of Gating the trivial trajectories A and B in Figaro 1 Ladner the trajectories A 

and B me obtained as a resah of the search procedare. the co ne sp onding strings of commands can be 

stored far sa b seqae nt using them as solutions in correspondi ng mkt [371. 


In the case shown m Figure lljbwc have a less trivial situation. Indeed, the lower kvd search can 
lead as in a straightforward manner to the trajectory A. H ow e ver, when the back-up motion is 
allowed, at least one additional trajectory should be added: a backing ap morion B and then motion 
forward C This new opport un ity immensely reduces the productivity of search. One would probably 
prefer to first select one of these opportunities (A or B+C) at the higher level, and then compute the 
actual trajectory. 



* • • * ngare 12. Taiwan rw tf »w* t« f an rrh AppMntlaa 

te m Nawad Mtra re k ra l C tkwB tf 


Firm 1L Faa mpt ra af —We typical i 


IF ROBOT IS BECOMING CLOSE TO AN OBSTACLE (DISTANCE IS LESS 
THAN D) 

(ON THE RIGHT, OR ON THE LEFT) 

THEN PUT MESSAGES TO PILOT AND EXECUTION CONTROLLER TO 1 
PRIORITY 
AND 

IF DISTANCE IS LESS THAN D* AND MORE THAN D** 

THEN REDUCE SPEED TO V* AND MAKE A TURN 

(LEFT, RIGHT) 

AND 

IF DISTANCE IS LESS THAN D** 

THEN STOP AND REPLAN 


One can see that these roles do not refer to any specific global coordinates of the trajectory to be 
executed, and the local coordinates (position of the obstacle relative to the robot) are given with low 
accuracy using linguistic variables as it was done in [23]. Similar roles are formulated for a more 
subde maneuver when after the local goal was determined in a particular situation, it turned out to be in 
an inconvenient location. e.g. the actual location of G~ Figure 6) cannot be achieved in a 
near- minimum time fashion by a simple continu at ion of the motion ahead. 

IF G(X) IS LESS THAN R T MIN 

AND G(Y) IS LESS THEN 2 R T M7N 
AND ORIENTATION IS LESS THEN |90“1 


THEN BACK-UP AND TURN IN THE DIRECTION OPPOSITE TO THE GOAL 
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( G(X) and G(Y) are the V and "y" coordinates of the local f oal in a local reference frame, ie. 
when the coordinate system is attached to the robot, R T min is the minimum radius of turn). 


Tis rule allows to perform a nontrivial motion shown m Figure 1 l,c which reproduces a very 
anth ropomorp h ic way of dealing with this probtenuNo wonder - it was introduced by a human 
"masrer^.We do not have any experience in formulating rules like this aotomarically.lt seems that 
using the human experience and intuition in formulating rules would be beneficial for dealing with 
predicaments arising during processes of operation. 

The overall process of using the rules previously formulated and stores, and the search when the rule 
is not found in the list, is illustrated in Figure 12. 

Learning of nrics.Thus, if the rule is not on the list, the recommendation is obtained using the 
search in the state space, or in other words, by solving the automaton of representation, or by solving 
difference equations of representation via (say) enhanced dynamic programming [31, 33, 36]. The 
results of this search can be stored and later used as a rule. Then the number of rules will grow 
tremendously, and instead of spending rime for search the system will spend rime for browsing the 
lists of rules. Probably, there exist an optimum number of rules depending on the descriptive 
properties of the space as well as the characteristics of the computer system utilized in a specific 
intelligent robot 

On the other hand, if a group of rules can be generalized and pushed to the upper level of the 
hierarchy, then the overall time of search can be reduced. Va% generalization can be done as follows 

DETERMINE FEATURES OF CONDITION PARTS OF RULES IN STORAGE 
FIND SIMILARITIES AMONG THE FEATURES OF RULES IN STORAGE 
GROUP THE CONDITION PART OF RULES IN STORAGE 
GROUP THE CONSEQUENT PARTS OF RULES WITHIN THE GROUP OF 
RULES 

WITH SIMILAR CONDITION PARTS 

SUBSTITUTE ALL CONSEQUENT PART BY ONE (AVERAGE) SOLUTION 
TO THE CORRESPONDING LIST OF RULES 

The process of learning with new rules generation is illustrated in Figure 13. The stage of 
"generalization" can be performed so far only with "master* direct participation. Indeed, the control 
strings found as a response to the concrete situations should be stored but the subsequent analysis 
should be done by a “master* unless the reliable algorithms of generalization are develops! 


ntw* 13. S*w lain i 



I 


Task from (i-l)-lh lev*! 


TO THE 
EXECUTION 
CONTROLLER 




In the meantime, the other types of learning (primarily, inductive leanring) can be cxcerciscd with no 
human involvement. They include memo riz ing the snapshots of the world, analysing the evolution of 
the snapshots, and computation of correct i ves for the matrices A and B as well as for the string of U. 

Retrieval of ruIes.The first experience of operation of autonomous mobile robot Drexel 
Dune-Buggy has demonstrated that us in the lists of rules is inefficient, and the hierarchical structure 
increases the productivity of the rules retrieval. The structure is shown in Figure 14. It is based upon 
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the theory presented in [38]. L^ is the language of conditions conutting of their vocabulary and the 
weighted relationship* among them. Ly h the language of consequents also consisting of the 
vocabulary and the gr ammar of weighted relationship* among the words of this vocabulary. The 
matching algorithm is based upon the grammar translator co nsistin g of the weighted relationships 
among the words of L* and Ly. Tme matching process C ends with obtaining several "best" 

machings, then evaluation, selection, and decomposition of the proper rule is done ({EJS£>}). 

After de c omposition is completed, focusing of attention should pe performed (FA), after this the 
search of the proper rule of the next level can be performed. The process is not fully automated since 
the process of focusing attention not always can be done with no master involvement. 

IV. Coodushms 

In existing systems the function of "master* is working in the DIM-aiea. This presumes that master's 
participation is expected to be required at the stages of 

context selection, 

arranging the blackboards for the specific context, 
assignment of the search envelope, 
supervised learning. 

To be more specific his functions will include (but not limited to) the following activities: 

1. Making available the "universe of discourse" knowledge. 

2. Proper selection of the context subset 

3. Monitoring the processes of image recognition especially while sensor integration. 

4. Final judgment on the space discretization (uniform, nommiform,etc) based upon master's 
opinion on what is "meaningful neighborhood". 

5. Monitoring the process of cost-function assignment 

6. Monitoring the process of vocabulary determining for the adjacent levels. 

7. Elimination of discrepancies in world representation at the adjacent levels. 

8. Monitoring the assignment of the relations of nesting by generalization. 

9. Determining the scope of attention according to the general assignment of the operation. 

10. Final decision on the duration of retaining the stored information especially for the levels of 
ephemeral maps. 

1 1 . Detenmnimg the rules of the high level of generality 

12. Arbitration among the multiple alternatives. 

13. Direct dealing with the problem of the lower level if the upper level vocabulary is poor. 

14. Participation in the processes of new rule recognition. 

15. Doing the job of linguistical tesselction: determining the groups of entities belonging to the 

same level of discretization. 

Expectantly, the language for communication with the "master" will be different from the language to 
be utilized for dealing with the other robots of the cooperative t eam 
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1. Abstract 

This paper describes a 3-D world Modeling technique using range data. Range data quantify 
the distances from the sensor focal plane to the object surface, i.e.. the 3-D coordinates of 
discrete points on the object surface are known. The approach proposed herein for 3-0 w>rld 
modeling is based on the Combinatorial Geometry (CG) Method which is widely used In Monte Carlo 
particle transport calculations. First, each measured point on the object surface is surrounded 
by a small sphere with a radius determined by the range to that point. Then, the 3-D shapes of 
the visible surfaces are obtained by taking the (Boolean) union of all the spheres. The result 
is an unambiguous representation of the object's boundary surfaces. The "p re- learned" partial 
knowledge of the environment can be also represented using the CG fethod with a relatively small 
amount of data. Using the CG type of representation, distances in desired directions to boundary 
surfaces of various objects are efficiently calculated. This feature is particularly useful for 
continuously verifying the world model against the data provided by a range finder, and for inte- 
grating range data from successive locations of the robot during motion. The efficiency of the 
proposed approach is illustrated by simulations of a spherical robot in a 3-D room in the presence 
of moving obstacles and inadequate pre-learned partial knowledge of the environment. 

2. Introduction 

An autonomous robot must have sensory capability to deal with unknown or partially known environments. 

The sensor derived data need to be processed to an appropriate internal representation of the external world. 

The external world to be described is fundamentally three-dimensional. Involving object occlusion. Most compu- 
ter vision research performed during the past twenty years has concentrated on using intensity images as sensor 
data. The imaging hardware (cameras) for these studies typically project a three-dimensional scene onto a 
two-dimensional image plane, thus providing a matrix of gray level values representing the scene from a given 
viewpoint. These values Indicate the brightness at points on a regular spaced grid and contain no explicit 
information about depth. Methods that use intensity information only for deriving 3-D structure are nsually 
computationally intensive. This computationally expensive processing arises due to the fact that correspon- 
dence of points between different views must be established and a complex system of nonlinear equations must 
be sol ved([l]-[5]) . 

In recent years digitized range data have become available from both active and passive sensors, *nd the 
quality of these data has been steadily improving([6]-[8]). Range data quantify the distances from the sensor 
focal plane to an object surface. Since depth information depends only on geometry and is independent of illu- 
mination and reflectivity, intensity image problems with shadows and surface markings do not occur. Therefore, 
the process of representing 3-D objects by their shape should be less difficult in range images than in intensity 
images. The problem addressed by this paper is the external world modeling using range data. Unique require- 
ments for such a model are: 

a) Allow representation of a general 3-D unknown or partially known environment, based on range data. 

b) Allow for minimal fast memory for storage. 

c) Allow the introduction of a feedback loop for continuous verification of the world model against the 
data provided by the sensor (efficient distance calculation), 

d) Allow for efficient integration of the range data from multiple views. 

e) Allow for efficient navigation and manipulation. 

A wide variety of techniques have been developed for representing 3-D objects for digital computing pur- 
poses. There are methods which describe the surface boundary and methods which represent the object's volume. 

The simplest boundary representation is using n-sided planar polygons (triangles, quadrilaterals, etc.) which 
can be stored as a list of 3-D node points along with their relationship information. Arbitrary surfaces are 
approximated to any desired degree of accuracy by using many planar polygons. This type of representation is 
popular because model surface area is well defined and all object operations are carried out using piecewise- 
planar algorithms. The next step in generality is obtained using quadric surface boundary representation. More 
advanced techniques for representing curved surfaces with higher order polynomials or splines are mentioned in 
the computer graphics and CAD Hterature([9]-[12]). There are many different techniques of this type; however 
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they art generally not vary awpact In terms of data storage. nor are they computational ly efficient in calculat- 
ing distances to boundary surfaces[13]. The best known volumetric representations are the oct-tree{14] general- 
Izad QrllnderClS] and multiple 2-0 projection v1tws[16l. The generalized cylinder approach Is well salted to 
many real world shapes. However, it becomes just about Impossible to use this representation for large, thin 
objects. The multiple 2-0 projection view Is not a general purpose technique since different objects may have 
similar 2-0 projections. The oct-tree representation Is used In many world models. However, the indexing 
prob1emCl7] Is seriously affecting the efficiency of this tecnnlque. In conclusion there is a need for a fast 
and robust technique for building 3-D models of arbitrarily shaped objects. 

In Chapter 3, a proposed external world modeling procedure and an efficient distance calculation algorithm 
are presented. A technique for Integrating the range data from multiple views and a continuous verification 
procedure of the world model versus the range data provided by the sensor is illustrated in Chapter 4. Finally, 
the feasibility of the proposed approach Is Illustrated in Chapter 5 by simulations of a spherical robot navi- 
gating in a 3-0 room In presence of static and moving obstacles and Inadequate pre-leamed partial knowledge of 
the environment. 

3. Representing 3-D Surfaces Using the Combinatorial geometry 

The basic problem addressed In this paper is one of representation. The proposed approach is based on the 
Combinatorial Geometry (CG) method[18] which Is widely used in Honte Carlo simulation of particle transport in 
3-D geometries. In CG (also known as Constructive Solid Geometry (CSG) In computer graphics and CAD litera- 
ture) solids are represented as combinations of primitive solids or "building blocks" (i.e., spheres, cylinders, 
boxes, etc.) using the Boolean operations of union, intersection and difference. The storage data structure is 
a binary tree where the terminal nodes are instances of primitives and the branching nodes represent Boolean 
operators. Any 3-D known object can be represented by a (Boolean) combination of primitive solids or deformed 
superquadrics[I9]. This representation is especially suitable for describing pre-leamed partial knowledge of 
the environment. An example of describing an abject composed of two boxes, one of them with a cylindrical hole 
Is Illustrated In Fig. 1. The result Is a concise, unambiguous and complete representation of the abject volume 
and boundary surface. 
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Fig. 1. Representing a 3-D object using Combinatorial Geometry, 
a - given ooject and its CG representation, 
b - the storage data tree. 


The result of a range scan is a matrix of distances from the sensor focal plane to an object surface. In 
other words, the coordinates of discrete points on the "visible* parts of the boundary surfaces of different 
objects in the external world of the robot, are known. Let oe the (swell) angle between two successive 
"reading" directions of the sensor. First, each discrete point i, is surrounded by a small solid sphere with a 
radius, r^ - aax(R 1 sin <*, a R^, where Rj is the range to point i, and ARj is the associated measurement error. 
Then, the approximate 3-0 shape of the visible boundary surfaces is obtained directly by taking the wiion of all 
the spheres (see Fig. 2). 

OftNL-DWC 86-16943 



Fig. 2. Describing the shape of 3-0 objects using spheres. 


The reason for using spheres Is to keep the representation as compact as possible. Describing the sphere 
for a particular discrete point In space means adding only one additional parameter (the radius) to t.*e coor- 
dinates of the discrete point which are provided by the sensor. The radius r< is defined as r^ • «u(Rj sin «, 
4*1 ) to avoid the appearance of "holes" in the geometry and to take into account the range uncertaixty. Using 
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tills definition for rj, neighbor spheres overlap one another and the boundary surface of the union of all 
spheres Is continuous (without holes) from the robot's point of view. Finally, It Is obvious that using the 
‘sphere* procedure, the shape of the boundary surfaces Is distorted. However, the distortion Is practically 
proportional to the range to each point since the range uncertainty 1$ relatively snail. In other words, the 
resolution of the node! Is fmp roved as the range to the surface Is decreased. 

3.1. Distance Calcelatlon In CG ^presentation 

A very useful feature of the CG representation Is Its efficiency In calculating distances to 3-0 surfaces 
In a desired direction. Observing discontinuities In the range data greater than the maximum size of the robot, 
the scene Is partitioned In many different zones. A zone Is defined as the union of snail spheres located 
between two successive discontinuities In the range data. Using the storage data structure Mentioned In Chapter 

3, two tables are defined: the first one includes the spatial location of the snail spheres; the second one 

Identifies the different zones In terns of these spheres. The distance to 3-fr surfaces In a desired direction 
fro* a given point. Is calculated in two steps: 

a) Each zone Is surrounded tightly by a box (rectangular parallelepiped). Since the boxes are approxi«ate 
bounding configurations. Intersections of a given ray with a box does not necessarily 1*ply Intersection 
with any particular sphere. In addition, the different orientations of the sphere clusters Imply that 
bounding boxes can Intersect. and therefore wftiple boxes may have to be checked for penetration by a 
given ray. The box (boxes) penetrated by the ray Is determined by calculating the Intersection points 
between the boxes and the ray. A list consisting of the boxes physically penetrated by the ray. Is 
defined. The corresponding list of zones is used to determine the penetration point. 

b) Determine the small sphere penetrated by the ray and calculate the penetration point. This is done by 
considering only the spheres Included ia the zones listed in the first step. Using this approach, only 
a small number of spheres are checked for penetration, and therefore significant computation time Is 
saved. 

It should be mentioned that the boxes surrounding the zones are used only internally during distance calcu- 
lations and they are not affecting the geometric description of the 3-D surfaces. During path planning, "tenta- 
tive paths" are checked for potential collision by calculating the distances to object surfaces from scattered 
points on the robot's surface in the desired direction. These distances can be effectively calculated by using 
the CG representation, and the procedure outlined above. 

4. Updating the World Model 

Automatic construction of 3-D models of objects fro* multiple views is an important problem in computer 
vision. In the past, a number of different techniques have been used for representation and modeling of 3-D 
objects for computer vision appl icatioas([20]-[27]) . However, there is an absence of a fast and robust tech- 
nique for building 3-D models of arbitrarily shaped objects. The process of constructing 3-D models for objects 
Involves integrating the range data from multiple views. In general, the integration process performs matching 
to establish correspondence between the views, determines the tnterframe transf ormat ions to register the views 
in a common reference coordinate system and then merges the data. The difficult and time consuming step in the 
above process is the notching step required to establish a correspondence. Much of the previous research efforts 
have been directed toward solving the difficult correspondence problem. The algorithm presented in this paoer, 
does not require any correspondence between different views, because the world model uses a universal coordinate 
system with the origin arbitrarily located at the robot's initial position. According to the representation 
algorithm described in Chapter 3. the accuracy with irfifch a certain point in space may be observed by the robot 
depends upon the distance between the robot and the point. This fact is translated to the radius of the sphere 
surrounding a particular point in the CG representation. Therefore, a point in space should be kept in memory 
along with the most accurate information (shortest observation distance). In other words, for each "measured 
point" in space, the shortest observation distance in the robot's history should be determined. The main problem 
in implementing this approach is the fact that since the sampling procedure of range data is discrete, the proba- 
bility of a particular point to be sampled from two different positions of the robot is zero. In other words, 
each "measured point" is sampled just once during the robot history. The solution implemented in our approach 
follows an iterative algorithm using the "old data" acquired before the current scan and the "new data" acquired 
during the current scan: 

a) The "old data" is checked from the current position of the robot. Using the world model based on the 
"old data", distances to 3-D surfaces fro* the robot's current position in the direction of points in 
the "old data" are calculated. If the distance to 3-D surfaces is smaller than the euclidean distance 
to the sphere surrounding the point then this particular point cannot be seen fro* the current position 
of t he robot and the point ^epresentat i on is kept unchanged. ?ther>ise. the "old* radius of the 
surrounding sohere is compared with the “new" radius determined by the euclidean distance from the 
current position of the robot and the smallest radius is chosen between the old and the new radii. 

b) The "new data" acquired from the current position of the robot is checked against the "old data". If 
the "new" point (provided by the sensor) is located within the world model based on the old data (within 
a sphere surrounding an "old" point) then the new point is rejected and therefore no new sphere is added 
to the world model . 
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If the “hew” point Is outside the old world node!* then the distance to 3-D surfaces In the "new" point 
direction Is calculated (using the old world Model). If the obtained distance Is greater than the range 
to the "new* point (provided by the sensor), the "new” point Is added to the world nodel as a sphere 
with a radius determined by the range to the point. Otherwise (the obtained distance Is snaller than 
the range to the point) the "old" point (located approximately In the same direction, but closer to the 
robot) Is erased froe the node) and the "new” point Is added to the mrld nodel. This Is the case of 
moving objects. In which the "old" data should be continuously verified and updated. 

c) Verification of pre-leamed geometric knowledge of the envir o nment. 

The sensor derived data Is compared with the calculated distances obtained from scanning the pre-leamed 
geometric envi ron ment. The pre-leamed data Is represented In a very concise way using the CG repre- 
sentation. If the ”rea1” range *n a certain direction Is found to be similar (within the uncertainty 
of the pre-leamed data) to the calculated range In the same direction, the representation is kept 
unchanged and no point Is added to the world model. If the real range Is snaller than the calculated 
range, the new real point Is added to the world model. Finally. If the real range Is greater than the 
calculated range, the entire pre-leamed object Is r emoved from the world model and the "real" point is 
added to the representation. 

5. Sample Problems 

The efficiency of the proposed world nodel Is Illustrated in several simulations of a spherical robot 
navigating In a 3-D room In presence of static and moving obstacles and Inadequate pre-leamed partial knowledge 
of the environment. The robot is assumed to move In a plane parallel to the floor, along straight lines. The 
origin of coordinates is arbitrarily located at the robot’s starting position. The goal coordinates are known 
a-priori. The external world geometry, the robot starting position and the goal location are Illustrated in 
Fig. 3. The radius of the spherical robot is 3 cm. The plane of motion is 30 cm off the floor. The navigation 
algorithm used in the sample problems is described in detail In Ref. 28. 

M emm m. ■«*. 


1 

1 

R - The Initial position of the robot (4.0.30) 


G - The goal position (190,0,30) 

B1 - BOX; dimensions (20 x 15 x 40); Center at (30,37.5,20) 

B2 - BOX; dimensions (10 x 60 x 90); Center at (-15.35.45) 

B3 - BOX; dimensions (40 x 20 x 90); Center at (100,30,45) 

C - CYLINDER; Center of basis at (60.0,0) 

Height 60, Radius 15 

S - SPHERE; Center at (110, -20. 30) 

Radius 20 

P - PRISM; dimensions (30 x 30 x 90); Vertex at (140.-10,0) 


Room dimensions: 200 x 100 x 100 

All dimensions are In centimeters 



Fig. 3. The geometry of the room. 


To illustrate the efficiency of the proposed technique for building the world model, four sample problems 
have been considered. In the first problem the 3-0 environment is completely unknown and the robot Is repre- 
senting the surrounding environment using the range data provided by the sensor. Figures *-8 illustrate the 
plane of motion during the robot's Journey from its initial position to a final position where he can directly 
"see" the goal. The world representation Is continuously updated using the information provided by the sensor 
from different reading positions of the robot. It can be seen that as the robot proceeds to the goal the world 
representation becomes more complete. 

In the second problem (Figs. 9-13), the box B1 and the prism P are provided a-priori to the robot (pre- 
learned knowledge). The robot is verifying the accuracy of the pre-leamed information and after finding It 
correct, is representing the two objects using two CG primitives (Box and Prism), without using the sphere type 
of representation. The representation of the overall 3-0 world is thus more concise than in the first problem, 
in which the sphere procedure was used to represent all objects. 
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The external world geometry considered In the two last problems Is similar to the geometry of the first two 
problem, except that the boxes Bi, 82 end the cylinder C ere rent if Iron the scene* In the third problem the 
box 83 end the prism P ere defined as pee- learned Information which ( Intentionally) was provided Inaccurately to 
the robot. Froe Figs* 14-17 It can be seen that the robot Is verifying the pro- learned data and finding that 
the box Is inaccurately positioned (Fig. 14) is using the sensor (real) data only to represent It (Figs* IS and 
16). At a later stage, (Fig. 17) the robot can directly check the pro- learned information for the prism (which 
was previously occluded) and finding it Incorrect Is removing the prism from arnwry. The prism fs then accu- 
rately represented using the data provided by the sensor. 

In the last example, the box and the prism are correctly provided as pre- learned information, and the 
"unknown* sphere Is moving forward and backward between successive positions of the robot. Figure 18 Illustrates 
the environment with the sphere at its initial position. While the robot is moving to the second position (Fig. 
19) the sphere is moving forward. The previous information about the sphere Is then checked, found Incorrect and 
r e mov ed from the robot's memory. Finally, Wien the robot Is reaching the next (third) position, the sphere has 
moved back to Its initial position. It can be seen (Fig. 20) that the robot is keeping the previous information 
about the sphere, since It is now occluded the "real* data and therefore cannot be verified. If at a later 
stage the robot is again In a position to directly "see" the "old" position of the sphere, this previous Infor- 
mation will be checked and eventually removed from the world model. 

These and the following figures shown in this paper have been produced using a computer printer and a very 
simple plotting routine. Since the maximum resolution of the printer along the Y axis (across the page) Is 130 
characters, certain existing spheres having diameters smaller than the printer resolution are not printed and 
therefore some "holes" may artificially appear on surfaces which are la fact continuous. 

6. Conclusion 

The proposed approach for modeling the external world using tne Cortlnatorlal Geometry was found promising. 
The range data from successive locations of the robot during motion can be effectively confelned and given an 
adequate world representation. The pre- lea mod knowledge and moving objects In the scene can be effectively 
verified and represented In the world model. The computation time per "picture* Including the simulated range 
scan, modeling the geometry, trajectory planning and plotting the plane of «>t1on was 30 s to 1 min CPU time of 
VAX-8800 Computer, depends upon the scene complexity and the number of tentative paths considered. More than 
SOt of the computation time Is used for plotting the pine of motion and for calculating distances In a given 
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direction from discrete points. These calculations can be executed Independently and therefore, performing the 
same calculations on a parallel or concurrent computer may significantly reduce the computation time. Future 
work using the proposed external world modeling approach will focus on the following Issues: scene segmentation 
Into objects, feature point extraction, recognition of 3-D objects from range data, replacing the sphere repre- 
sentation with a more concise C6 volumetric representation of the recognized objects and finally Implementation 
of this method on the NCU8E Michlne and experimental verification using the HERMIES-11 robot. 
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ABSTRACT 

In this paper we consider the following problem: A point 
robot is placed in a terrain populated by unknown number of 
polyhedral obstacles of varied sizes and locations in two/three 
dimensions. The robot is equipped with a sensor capable of 
detecting all the obstacle vertices and edges that are visible 
from the present location of the robot The robot is required 
to autonomously navigate and build the complete terrain 
model using the sensor information. We establish that the 
necessary number of scanning operations needed for com- 
plete terrain model acquisition by any algorithm that is based 

* 

on ’scan from vertices’ strategy is given by (<>,)-« and 

i*l 

m 

£N( Oi)-2n in two and three dimensional terrains respec- 
i*1 

tively, where O -{0 ■ * ,0* } is the set of the obstacles in 

the terrain, and N(O t ) is the number of vertices of the obstacle 

Keywords and Phrases: 

P3th Planning, Terrain Acquisition, Collision Avoidance. 
I. INTRODUCTION 

In recent years there has been an enormous amount of 
research activity generated in the area of navigation and path 
planning for mobile robots. Much of this work could be 
thought of as an offshoot of the pioneering works of Lozano- 
perez and Wesley [1], Reif [2], Schwartz and Sharir [3], and 
O’Dunlaing and Yap [4]. In this work the robot is located in 
a terrain whose model is precisely known. A path ha to be 
planned to navigate a robot from a specified point to a 
specified destination point (if such path exists). A 
comprehensive survey of these and related techniques for 
robot path planning is available in Whitesides [5]. Another 
important problem is the navigation in unexplored terrains . 
Here the robot is equipped with a sensor with which the robot 
scans the terrain, and a navigation jjaih is planned based on 
these sensor readings. In general several sensor operations 
are needed for planning a navigational course. Lumelsky zrd 
Stepanov [6] present nice solutions to a restricted version of 


this problem. Iyengar et aJ [7] and Rao et al [8] present a 
technique that utilizes the sensor readings to construct a 
world map through incidental learning, Oommen et al [9] 
presents a more formal treatment for the case of convex 
polygonal obstacles. In these approaches the terrain model 
acquisition is purely incidental i.e., the construction of the 
terrain model is only secondary and scanning is performed for 
the purpose of navigation. 

Another important problem in the navigation in unex- 
plored terrains is the Terrain Acquisition Problem in which 
the robot is required to autonomously navigate and build the 
complete terrain model through the sensor readings. In this 
paper we consider the following version of terrain acquisition 
problem: A point-sized robot M is placed in a two/three 
dimensional obstacle terrain 0 . The tenain 0 is populated by 
the set of obstacles {0 X ,0 2 , * * *0,}, where 0, is a 
polyhedron. We assume that 0 is finite, i.e. f 0 can be 
inscribed in a circle/sphere of finite radius in two/three 
dimensions. Furthermore each 0, is finite and had a finite 
number of vertices. Initially the sizes and locations of the 
obstacles are totally unknown to the robot. The robot M is 
equipped with an ideal sensor system capable of detecting all 
edges and vertices visible to the robot from its current posi- 
tion. The robot is required to autonomously navigate in the 
terrain and acquire the complete obstacle tenain model, i.e. 
obtain the locations of all edges and vertices of each obstacle 
of 0 . The main motivation for this problem stems from the 
fact that after terrain acquisition phase, the future navigation 
of the robot can be carried out without sensor operations 
using the techniques for navigation in known terrains. In 
many cases navigational path can be made optimal in terms of 
the distance to be traversed by the robot. 

A solution to this problem is given by Rao et al [10] 
based on the incremental construction of the visibility graph 
of the terrain. The same technique is extended to a finite- 
sized robot in plane by Rao et al [1 1]. The algorithm of [10] 
is guaranteed to acquire the complete terrain model in finite 
time. The algorithm terminates when a scan operation is per- 
formed from each vertex of every obstacle and consequently 
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II 

the number of scanning operation required is £ //(O, ), where 

i« 1 

N(Oj) is the number of vertices of the obstacle O t . However, 
this is only a sufficient condition on the number of scan 
opera t io ns . Ln this paper we establish that for any terrain 
acquisition algorithm (based on scan from vertex strategy) 
there exists a terrain O such that the necessary number of 

» m 

scan operations is given by £N(0i)-n and £N(0i)-2/i 

i»I i* 1 

respectively for two and three dimensional terrains. In other 
words, no more than one (two) scan operations per obstacle 
can be skipped in two (three) dimensional terrains. We also 
show that a strategy that randomly skips one vertex (two ver- 
tices) per obstacle will not acquire the complete terrain model 
in two (three) dimensional terrains. We then list a number of 
issues for future research. 

The organization of the paper is follows: In section 2, we 
briefly discuss the issues involved in the terrain acquisition 
problem and also the algorithm of [10]. In section 3, we 
present the bound on the necessary number of scan opera- 
tions. 

2. TERRAIN ACQUISITION METHODOLOGY 

During the terrain acquisition the robot M is required to 
plan and execute a navigational course; robot stops at certain 
points, called the sensing points , on the path to carry out the 
scan operations. The terrain model is reconstructed by 
integrating the scanning information obtained from the indivi- 
dual scan operations. In general, the navigational path could 
only be planned in an incremental manner by utilizing the 
scan information because the terrain is unexplored. The main 
requirement on the terrain acquisition algorithm is that the 
complete terrain model should be acquired in a finite amount 
of time. 

Here we deal with vertex-based terrain acquisition 
methods where the sensing points are always vertices of the 
obstacles, i.e.. every scan operation is performed from an obs- 
tacle vertex. The robot M moves from vertex to vertex dur- 
ing the navigational course. The algorithm of [10J is based 
on this strategy. There are two key issues that are important 
for a terrain acquisition algorithm: 

(a) Computing the next vertex to be visited, 

(b) Detecting the completion of terrain acquisition (termina- 
tion of the algorithm). 

We now bnedy discuss the terrain acquisition algcnthm 
of [10]. Let VERiPi) denote the set of vertices of O t . Let 

m 

V=\jVER (O t ) be the set of all vertices of the obstacles. The 

visibility graph of the terrain 0 % denoted by VG(O), is a 
graph (V,£), where an edge ^.exists if and 

only if (v^vj is either an edge of an obstacle or V! is visible 
from v 2 and vice versa. In Fig.l. an obstacle terrain popu- 
lated by three obstacles 0 { .0 2 and is shown and its visi- 



Fig. 1. Obstacle terrain 

bility graph is shown in Fig.l A vertex is said to be explored 
if a scan operation is performed from v , and otherwise v is 
said to be unexplored. Once v is explored then the adjacency 
list of v in the visibility graph is known. The robot M is ini- 
tially placed at a point in the obstacle terrain. Then M scans 
and moves to a vertex. From this point the terrain acquisition 
algorithm, called algorithm ACQUIRE, of [10] is invoked. 
Let M start at vertex v#eV. A scan is performed and the 
adjacency list of v 0 is stored Then M moves to an adjacent 
unvisited vertex and recursively applies this method When 
an unexplored vertex is visited it is poshed onto a stack called 
path-stack . Let M be located at a vertex v from which it per- 
formed a scan operation. Then M moves to a nearest unex- 
plored vertex adjacent to v if one exists. The M can move to 
this chosen vertex in a straight line because it is seen from v. 
If all the vertices adjacent to v are vived then the path-stack 
is used to obtain the next sensing point. The top of the path- 
stack is recursively popped till a node v x with unvisited adja- 
cent nodes is found. Shortest paths to all the unvisited adja- 



Fig. 2. The vjsibiliry gr3pti for ihe terrain of Fig. 1. 
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(a) Navigational path (shown in dart) 
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In the next section we show that for any vertex-based 
terrain acquisition algorithm there exists a terrain such that 
the necessary number of scanning operations is given by 

ZMO.h. 

• •i 


3. NUMBER OF SCAN OPERATIONS 

Consider a vertex-based ten din exploration algorithm 
(and algorithm of [10] is one such). The algorithm performs 
scans and detects newer vertices which will be explored in 
subsequent scans. During terrain exploration by a vertex 
based algorithm no more than one vertex per obstacle can be 
left unexplored in ttfo dimensional terrain constructed as 



(b) Partially built visibility graph 


Fig.4. Intermediate stage of exploration 


explained below. For three dimensional terrains no more than 
two vertices per obstacle can be left unexplored in our spe- 
cially constructed terrain. The basic idea is illustrated in 
Fig.5. We consider a single convex polygonal obstacle in 
Fig J(a). If M starts at a vertex it detects one new vertex with 
one exploration (except when the first vertex is explored) of a 
vertex as the robot moves along the circumference of the 
obstacle. In other words at no point of time the terrain 
acquisition could be declared complete if there are two unex- 
plored vertices say v x and v 2 . This is because the robot does 
not, in general, know what lies on the hinder (unexplored) 
side of the line joining v x and v 2 . There could a single vertex 
or a number of edges on the other side of the line joining v t 
and v 2 as in Fig J (b) and (c). For three dimensional terrains, 
m more t u ~r two ve nice* per obstacle can be left unexplored. 
This is because if three vertices (say v lf v 2 and *3) are left 
unexplored then the information on the hinder side of the 
plane formed by the vertices v lf v 2 and v 3 is net known in gen- 
eral. The hidden side of the obstacle can be either a simple 
plane or composed of a a number of planes as shown in Fig.6 
(a) and (b). 





(C) 


Fig.5. Two dimensional case 

Theorem 1: 

For a vertex-based terrain acquisition algorithm and 
given positive integer n there exists a terrain 
{0 U 0 2 , * ' * ,O m } of n polyhedral obstacles such that the 
necessary number of scan operations is 

Y,N(jOi)-n for two dimensional terrain 
•*t 

£N(Oi)-2n for three dimensional terrain 

i-1 

Proof: We use induction on the number of obstacles in 
the terrain. Consider «=l. In two dimensional terrains 
consider a convex polygon as in fig 5(a). Note that from 
a vertex v 2 , we can only see two vertices that are adja- 
cent to v. Apart from the first scan, no more than one 
unexplored vertex can be seen in any scan operation. 
From the discussion above M has to carry out sca nnin g 
till no more than one vertex is unexplored. Thus 
N(O x y\ is the necessary number of scan operations for 
two dimensional terrains. By similar arguments we can 
show that the necessary number of scan operation is 


Fig.6. Three dimensional case 


N(0 j)-2. Hence the claim is true for n »1. 

Assume that the claim is true for a *4. There exist a ter- 
rain of k obstacles with the necessary number of scan 
operations given in the theorem. Now const r uct a terrain 
of k+1 obstacles as follows: In two dimensions add a big 
polygon O m outside the circle inscribing the terrain that 
satisfies the induction hypothesis as shown in Fig.7. The 
fc+lth polygon has a long edge joining v x and v 2 that 
obscures the remaining edges of the polygon from the 
scan operations carried out in the terrain of k obstacle. 
Thus the scan operations needed during the exploration 
of the i+lth obstacle is V(O t+l >-l. Hence total number 
of necessary scan operations for two dimensional ter- 

rains is given by 2W(0jM*+l)- For three dimensional 

i*l 

terrains the obstacle is such that a plane formed by 
three vertices v lf v 2 and v 3 obscures the rest of the obsta- 
cle from a scan in the terrain of k obstacles as in Fig _8. 
The 0* + | lies outside the sphere the encloses the terrain 
of k obstacles. Using the arguments similar to two 
dimensional case we can show that the n eces sary number 
of scan operations to acquire is N{O k +i)-2. Thus 
the theorem follows by mathematical induction. □ 

In the above theorem we have seen that no more that 
one (two) vertices per obstacle can be left unexplored in two 
(three) dimensional terrain. The natural question is to ask if 
we can always skip one (two) vertices per obstacle for two 
(three) dimensional terrains. The answer is no as the vertices 
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Circle corn* :*.ing k obstacles 
Fig.7. Two dimensional case - Addition of O k +\ 
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sphere containing k obstacles 
Fig.8. Three dimensional case - Addition of O k + x 



Fig. 9. Configuration - two dimensional case 



Fig. 10. Configuration - three dimensional case 

are to be randomly skipped. This is illustrated in Fig. 9 and 
Fig. 10. In two dimensions the if the robot skips the vertices 
v lt v 2 and v 3 then the obstacle 0 4 will not be detected. Fig.10 
shows a three dimensional example. The configurations such 
as shown in Fig.9 and 10 can be formed with any (finite) 
number of obstacles which could be other than triangles or 
tetrahedrons. Fig.ll shows one such example. It is open at 
this point to design a vertex-based terrain acquisition algo- 
rithm (or show algorithm does not exists) that skips one (two) 
vertices for each obstacle and guaranteed to acquire the com- 
plete obstacle terrain model. 



4. CONCLUSIONS 

In this paper we have shown that for any vertex-based 
terrain acquisition algorithm there exists a terrain such that 
the necessary number of scan operations is given by 

« m 

JN(Oj)-l and £N(Oi)-2 respectively for two and three 

1*1 im\ 

dimensional obstacle terrains. In other words, we do not 
expect to design a vertex-based terrain acquisition algorithm 
that has complexity lower than the above sated sums (in 


287 



lent of the senior operation*).'!*®* exists a terrain acqmsi- 
bon algorithm with tie number of senior operations given by 

£N(0i)(lO]. It would be inte res tin g to see if there exists a 

M 

terrain acquisition algorithm with only the necessary 

number of sensor operatio ns given in this paper. 
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I) introduction: 

The SOLON Planner provides an architecture for effective read-time planning and 
replanning for an autonomous vehicle. The acronym, SOLON, stands for. State-Operator LOgic j 
MachiNe; the highlights of the system, which distinguish it from other Al-based planners that j 

have been designed previously, are its hybrid application of a state-driven control 
architecture and the use of both schematic representations and logic programming for the 
management of its knowledge base (1). 

SOLON is designed to provide multiple levels of planning for a single autonomous vehicle 
which is supplied with a skeletal, partially-specified mission plan at the outset of the 
vehicle's operations. This mission plan consists of a set of objectives, each of which win be 
decomposable by the planner into tasks. These tasks are themselves comparatively complex j 

sets of actions which are executable by a conventional real-time control system which does I 

not perform planning but which is capable of making adjustments or modifications to the 
provided tasks according to constraints and tolerances provided by the Planner. j 

sf i^Oflr current implementation of the SOLON is in the form of a real-time simulation of the 
Planner module of an Intelligent Vehicle Controller (IVC) on-board an autonomous underwater 
vehicle (AUV). The simulation is embedded within a larger simulator environment known as 
ICDS (Intelligent Controller Development System) operating on a Symbolics 3645/75 computer 
( 2 ). 

II) Real-Time Operational Context 

Many planning systems have been developed over the years of Al research, but few 
have been built expressly for use in the control of an autonomous vehicle which will be 
operated in extremely remote, inaccessible environments (3). Most autonomous vehicles to 
date have been land-based, and the focus of research has been upon problems of mobility and 
obstacle avoidance. We recognize that the AUV problem includes all of the same issues faced 
in the design of any autonomous vehicle but in our case long-range goals dictate the need for 
modularity and interchangeability of system components (e.g., sensor subsystems, 
manipulators, photographic equipment, etc.) and the integration of these subsystems with the 
controller responsible for moving the actual vehicle. Furthermore, an onboard IVC must be 
capable of performing in real time, without the typical options that are generally available to 
teaestial vehicles. An AUV has additional directional degrees of freedom than a terrestial 
vehicle as it moves through 3-space but it has less operational freedom to stop or to undo 
certain motions. For example, with the exception of certain experimental-only vehicle 
architectures, an AUV cannot retrace its path exactly and bringing the vehicle to a halt 
cannot make use of braking systems or high degrees of friction. 

The underlying philosophy of the SOLON planner includes the following basic premisses: 

i) Both the knowledge base and the work (the processing) must be distributed, allowing 
for concurrent processing by different components, in order to solve the real-time bottleneck; 

ii) The architecture must be dynamic and flexible to alow for the fact that different 
environmental and decision-making situations wil require a different balancing of the 
processing loads; 

in) Any intelligent controller must evolve in its design from the simple to the complex 


and that vehicle-specific and mission-specific requirements which are not presently 
foreseeable should not require major restructuring of a system design or implemented, operable 
software. 

Id) High-Level Architecture 

SOLON is composed of five logical processes (logprocs) which operate as cooperative 
but autonomous agents in the Planner. These are: 

i) Strategist - central decision-making; selection and scheduling and detailing of 

objectives and tasks within objectives; 

ii) Event Assessor - processing of events and condffions reported by other IVC 

modules about outside environment or internal vehicle systems; 

iii) Tactician - interfacing between Planner and lower-level vehicle control systems 

responsible for driving actuators and effectors; 

iv) Plan Simulator - evaluating alternative plans in cases where Strategist’s 

first-cut logic cannot derive a dear and distinct best choice; 

v) Plan Registrar - central editing and modification of the Active Plan. 

Each logproc communicates with others by means of an message-passing system which 
is also the structure employed for inter-modular communications within the whole IVC. The 
logprocs may be multiple processes which share a single-processor machine such as the 
Symbolics or they may be implemented in a distributed processing environment where each 
logproc has its own physical processor. Their relationships are described in a data flow 
diagram ( Figure 1) and their functions are described in more detail later in this paper. 

IV) Distribution of Intelligence and Dedsion-Maklng 

A major stumbling block to the implementation of real-time Al systems has been in the 
area of performance. Typically a system is bound with achieving a particular goal and 
events occur which cause one of the following types of situations: 

i) The original problem that the system is working upon is no longer relevant 

ii) The data being used by the system in its current problem-solving has been changed 

ifi) A new problem has higher priority and should take precedence over the current 

activities of the system. 

This is more than a problem of conflicting goals in which the system must decide to 
satisfy one. Rather, we are faced with goal conflicts that may not arise until after a given 
attempt to satisfy a goal is underway and which may not be communicated to the system to 
satisfy the goal. The cost in time and computation resources for determining how to alter 
the current activity and how to respond to a new situation may be prohibitive given the 
other requirements of the control system. 

The approach undertaken in SOLON distributes the work among the multiple logical 
processes which can proceed independently. The performance of the Plan Simulator is not 
impacted by the Strategist, which may be in an idle state or else busy running PROLOG 
inferences, other than by the constraints imposed by Ihe scheduler if both logprocs are 
running off the same physical processor. Obviously there is signficant advantage to a logproc 
having its own processor. At any given time, messages can be sent between logprocs in 
order to interrupt or provide additional information. There are a finite number of message 
types potential-obstacle, task -completed, relevant-object task-status, etc.) which may be 
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sent between logical processes but the schema-based representation structure allows for this 
number to grow as the complexity of the Planner implementation increases. 

When a message is sent, for instance, from the Tactician to the Strategist, the 
message is sent with the priority that has been assigned by the sender. This indicates how 
significant or urgent the sender believes the message to be. That wil not necessarily be the 
way the message is treated by the receiver, which may be working on a task that is the 
result of a previous message and which may be rated with a much higher priority. 

When the receiver gets a message, it is not automatically interrupted from its work. 
Instead, each process follows a cycle of execution followed by checking for messages. This 
has been implemented for the Strategist and is described further in section X If the receiver 
is at a point where it can be interrupted, then it wB determine if there is any message in its 
queue that demands a jump to a new task. The receiver process uses the Process-Msg 
function to determine what message is the most critical one in the queue. Currently 
Process-Msg is a LISP function that employs several simple rules but in future implementations 
it will be a compact expert system in its own right within the Planner. 

What becomes of the work that a logproc was doing if and when it is redirected by a 
message to tackle some other task? There are two possible approaches. One is to interrupt 
the original task, save a history of what was going on and possibly resume it at a later time. 
However, this poses many problems in truth maintenance and non-monotonic reasoning 
because the situation that gave rise to the original task may be altered as a result of the 
new operations being executed. Rather than attempt to keep track of all changes and 
determine that the original task should be resumed or not, the approach in SOLON is to simply 
drop the original task altogether. If in fact the original task should be resumed, then the 
conditions which led to that task will surface again in the form of a message from one 
logproc to another. Otherwise the system win deal with the highest priority messages that 
currently exist The Planner is always responding to the latest most current state of 
affairs, even if this means doing some extra work or duplicating a few steps to determine 
what should be done next. In other words, one may ask the question "What is the most 
pressing business right now?" more often, but one wil always have the best and most 
up-to-date answer to that question, given the rules that have been built into the Planner 
through its functions and specific PROLOG axioms. 

VI) Multiple Forms of Knowledge Representation 

Factual knowledge of the environment and the vehicle is maintained in a database 
which employs the schema-slot-value structure as implemented in Knowledge Craft, a 
Common USP based system tool developed by Carnegie Group, Inc. There is a fundamental 
taxonomy of world objects and relations’ which hold between various objects, as indicated in 
Figure 2. Generally, knowledge about "what is" is stored in this schematic representation, 
whereas "how to" knowledge, rules and operational guidelines, is represented in PROLOG 
axiomatic expressions. The PROLOG mechanism was deemed to be suitable for making queries 
and deriving solutions that could best be found using backward-chaining inference processes. 
However the PROLOG form was not deemed sufficient to be used as the only form of 
knowledge representation. 

The power of the schematic representation is that the 'sets are easily stored in a 
wet-structured format and the information is easily accessible. Moreover, from a 
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FIGURE 2 - Fundamental Otoj#ct* and Rttatfom In th* SOLON Woild 
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experimentalist point of view it is a good representation for buikfng a prototypical system, 
one that is open-ended and intended to evolve beyond one’s current expectations. 

Vehicle missions consist of human-spedfied skeletal plans, the elements of which are 
known as objectives - high-level types of operations such as transiting to a destination, 
searching a region for a given set of objects, folowing a moving object, etc. Objectives are 
deocmposabie into tasks, more elementary types of activity (transit to waypoint hold 
position, send transmission, manipulate arm or sensor platform, etc.). Mission specifications 
are, therefore, short program-tike structures that provide the Planner with a first-cut set of 
objectives to perform in a given temporal sequence with some parameters of those objectives 
detailed and others left for the Planner to address at run-time. The sequence of objectives 
and the parameters (e.g., bounds of a region to search, starting position for a search 
operation) are the default or first-choice values for the Planner to use as optimal guidetines. 
Through the built-in logic of the Planner and rules that are specified within the mission (see 
next section) the sequence and parameters and the choice of tasks for satisfying each 
objective are modified in response to events that occur both in the external environment and 
within the vehicle. 

VII) Hierarchy of Operational Rules 

A considerable database of operational rules is implemented using PROLOG. Certain 
rules are applicable to any mission for any type vehicle and these may be considered the 
most basic planning rules, modification of which constitutes a redesign of the planning 
algorithms. These rules are stored in a structure called Act-Prolog and are always present in 
the working set of PROLOG axioms available for inferendng operations. Other rules are 
specific to the vehicle but apply to any mission that such a vehicle might undertake. They 
are known as Sys-Oper rules and will also be available during the entirety of a given mission 
but may be replaced at pre-deployment time without disturbing the Act-Prolog body of rules. 
Obviously, this type of partitioning is principally for the benefit of easy system maintenance 
and modification and does not affect the actual Planner operations. 

A similar body of rules are global for a given mission and are accessible at ail times 
during the Planner operations for any objective that is a component of the mission. Next 
there are three classes of rules which are not accessible uniformly; these include: 

i) rules that apply during a specific type of mission objective; i.e., a special rule for 
search operations, which does not apply to transit or escort operations; 

ii) rules that apply for a specific type of task within an objective; i.e., a special rule 
that applies to 'waiting’ tasks and does not apply to motion-oriented tasks; 

iii) rules that apply for a specific objective within a mission; i.e., a special rule for 
when the vehicle is searching area B for a sunken object and only for that objective, having 
no applicability to other objectives within that mission. 

The purpose of this sutxSvision of rules is to allow efficient entry of new rules into the 
system and to manage the PROLOG inferendng process so that the system never has to deal 
with more than those rules which could possibly apply at a given instance. The aim is to 
reduce the size of the rulebase wherever possible, reducing the number of rules that must be 
examined and eliminating from consideration those rules that could not possibly have any 
relevance at a given instance. 


VHIjTIm Strategist 

The Strategist is dearly the most complex logical process within the Planner, and it is 
within the Strategist that the state-operator architecture is employed most fully. Ris also 
the logical process that makes most extensive use of the message processing and evaluation 
logic which is a critical element in the SOLON design. Messages which arrive from other 
Planner logprocs are processed according to an algorithm which attends to both the sender's 
and the receiver’s evaluation of the message priority and significance for the current state of 
the vehide’s mission plan. The algorithm we have initially employed is a simplified version of 
what we exped wil evolve into an expert system in its own right a dear place-holder for 
future machine learning studies. Depending on the message chosen to address, the Strategist 
moves into one of several state-operator functions or processes (currently all are 
implemented as LISP functions) and future actions of the Strategist are then governed by 
two factors: 

i) The results of function evaluations and hypothesis generations within the 
current state-operator; 

ii) The appearance in the Strategist’s message-queue of a 'critical message" 
demanding immediate attention and the overriding of current replanning activities. 

Examples of current state-operators indude: 

i) Anaiyze-Next-Scheduled-Objective 

ii) Expand-Objective- Into-T asks 

iii) Examine-T ask-Status-Msg 

iv) Determine-Plan -Change- Directive 

Within each state-operator, the main activities consist of determining what is the 
appropriate hypothesis to test and then making queries into the PROLOG-based knowledge 
base. This consists of a dynamic set of axioms (modifiable facts and constant rul9S). 
Through the PROLOG mechanism employed within our implementation (CRL-PROLOG) these 
have full access to the major body of represented knowledge about the vehide, mission plan 
and environment, which is in the form of schema-slot-value-relation data structures. 

The basic algorithm of the Strategist top-level function is presented in Figure 3. 

FIGURE 3 - Strategist Top-Level Algorithm 
(Outer loop] 

IF (Critical Message Received) 

THEN (Make the Critical Message We Cuner* Message) 

ELSE (Get toe Most Relevant Message From the Queue inhere is one) 

Based on toe type at message (related-event, obstacle. task-siMus. etc.] 
received. (Select Appropriate State-Operator To Activate) 

IF (Strategist is in waft-sta te) 

THEN (Put Strategist process into wait state) 

ELSE 

(Inner loop) 

UNLESS (Strategist is (traded into wait-state) 

OR (Critical Message Received) 

(Execute the Selected State-Operator) 
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IX) A Working Example 

The operation of the Strategist may best be described through the use of an example 
scenario which will illustrate the manner in which the state-operator functions and PROLOG 
rules operate to control the vehicle and replan its mission. We will consider a simple search 
mission as described by Figure 4. The tentative path of the AUV, were it to execute each of 
the initial scheduled objectives in sequence and without any replanning due to new events, is 
indicated by Figure 5. 

Initially the Strategist is given a start-mission message which activates the 
state-operator Analyze-Next-Scheduled-Objective. The obvious first step is to consider 
what is next on the list of scheduled objectives given by Mission Control. A series of Prolog 
queries are generated to determine if there are sufficient reasons for undertaking this 
objective and no overriding reasons to avoid this objective at this time. 

Assume that the first objective is to move the vehicle to a given point A. This 
objective is represented as Trans'it-A and it inherits all the default characteristics 
associated with transit operations, as well as any special rules that may have been specified 
by Mission Control in the skeletal plan. These defaults include the decomposition of the 
objective into component tasks. For point-to-point transits, there is only one elementary 
task, that of moving the vehicle in 3-space. However, before asserting a new transit-type 
current-task, checks are made to determine that there are no known obstacles (which may 
have been detected by the sensors) in the vehicle’s path. Using an algorithm which treats 
potential obstacles as expanded spheres and represents the vehicle as a point, lines of 
tangency are computed which provide possible new waypoints for the vehicle to use in 
navigating around the obstacles. An elementary set of rules determines which is the best set 
of alternative paths and these paths become the new component tasks for the current 
objective. 

Suppose that the vehicle is now engaged in a search objective, where there area 
specified number of objects (cylinders or curved surfaces with a radius > 3m but < 10m) which 
are deemed relevant to the search and one particular object (a pipeline on the seafloor) 
which is the goal object for the search. Specific rules provided with the mission plan indicate 
the actions to be taken if and when various relevant objects are detected. These are in the 
database of rules which are active while the vehicle is executing its search objective. 
Having received a message about a cylindrical object at point A from the World Modeller, the 
Event Assessor determines that the object is relevant to the current objective and the 
Strategist is notified. The latter responds by determining an appropriate change that applies 
to the current state of the Active Plan. This response, based upon rules input with the 
mission specification, may be to initiate a new objective, a spiral search operation centered 
upon the newly discovered object Before actually changing activities, the Strategist invokes 
a state-operator which explores "what if type queries to determine if instituting the change 
of plan would cause conflicts for other objectives of the mission. If there are significant 
conflicts, then the Strategist returns to the state-operator charged with determining an 
appropriate change of plan given the new event (object). Otherwise, the new objective is 
treated as the next scheduled objective (as if it were part of the original schedule) and the 
state-operator charged with checking out all next-scheduled objectives is then activated. 


FIGURE 4 - Search Mission Scenario 

SCENARIO-1 Test Mission— 

TRANSIT-A (movo vshido in straightast possible path to point (200 200 200)) 

LADDER-SEARCH-A (starch 200m x 200m region for sunken submarino vassal, 
using ladder-like motion pattam; relevant objects indude smooth, carved 
surfaces and metallic objects) 

TRANSIT-B (move from end of search operation to point (1200 1200400)) 

LADD ER-SEAR C H-B (search 400m x 400m region for same submarine) 

TRANSTT-C (move to point (2000 800 800)i 

SPI RAL-SEARCH-C (search 600m radius region starting from current position, 
moving counter-clockwise in a spiral motion pattam) 

TRANSIT'D (move to point (400 60 0)) 

REPORT-TO-BASE (transmit data from surface using radio) 


FIGURE 5 - AUV Scenario Path 

The world is a 3 km by 3 km square of coastal ocean 
of varying depths. 



(Y) The mission is to search for a sunken submarine 

in this hypothetical ocean space. 

The in' . ! il mission specification given to the AUU consists of eight 
objectives: Transit-A (go to ptA), Ladder-Search-A (search an area 
using a ladder-like motion pattam), Transit-B, Ladder-Search-B, 
Trensit-C, Ladder-Saarch-C.Transit-D and Raport-To-Base. 
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When the next scheduled objective 'checks out* as being satisfiable and not in conflict 
with other objectives (according to the rules provided for determining conflict and 
satisfiability), a state-operator is activated which 'expands" the objective into appropriate 
component tasks. In the case of transiting operations, these tasks will generally all be 
motion-oriented, but in the case of a search objective there may be an interspersing of 
(Afferent types of tasks among the motion segments required to move the vehicle around. 
These include performing intensive scanning of a region while the vehicle is at an intermediary 
waypoint in the search, manipulating objects if the vehicle is equipped with appropriate 
devices, and so forth. Once the new current objective has been 'expanded" into an initial set 
of tasks, these tasks are sequentially processed in a similar fashion - the next scheduled task 
is 'checked out" just prior to execution for consistency with the actual state of affairs 
(which may have changed significantly since the objective was expanded into a list of tasks) 
and if the new task is approved, it is then transmitted via the Tactician to the Vehicle 
Operating System, a control program whose function is, akin to the mechanical engineering 
staff on a ship, to carry out the high-level commands and operate the servos and actuators 
of the vehicle. Communications from the Vehicle Operating System back to the Planner 
consist of messages indicating the status of the given task - either that it has been 
completed or that it cannot be completed, given the constraints specified (e.g., maximum time 
to perform a transit, maximum deviation from a given course). 

Fundamental to the SOLON architecture is this partitioning or distribution of jobs 
among many different agents, the state-operator functions (4) of the Strategist. Many 
independent specialists, as it were, handle their particular tasks, without burdening each 
other or the higher-level modules in the Planner. However, with the message-passing system 
that runs throughout SOLON, the higher-level modules, like the captain and officers of a ship, 
have access to the activities of the lower-level units and are able to make changes which 
can include changing the tasks of those lower, simpler units. 


X) Conclusion 

The SC' Planner provides several new features which we believe are important for 

planning anc i ting, particularly in a real-time mobile context. First, it provides a 

mechanism fc ~ ..touting or partitioning the knowledge required for high-to-medium-level 
vehicle control into a number of different representation schemes (rather than just one 
method) and it to a number of independent but communicating databases. Secondly, SOLON 
provides a mechanism for distributing the work of evaluating alternatives and selecting 
sequences of objectives and tasks among several agents (logprocs) which can readily be 
implemented in a concurrent. MlMD-type machine architecture. Thirdly, SOLON operates by 
'default reasoning' principles - the network of state-operators is such that problem-solving is * 
attempted first using the simplest, most probable or most expected searches and queries. 
When the default methods fail, more complex logic is invoked. There is a definite redundancy 
built into SOLON; certain logprocs will receive messages and initiate activity which may turn 
out to be not required because of solutions implemented by another logproc. (This is 
particularly true in some cases of obstacle avoidance). This redundancy, we feel, is not only 
admissible but important The justification for such mechanisms and for a good part of the 
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state-operator architecture lies not only with the failure of many previous planning systems 
but the obvious historical success of biological planning in humans and animals. 
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Problems, Southampton Univ., England, April 1986 

(4) In our design the state-operators are entities represented using the common 
schema-based structure employed throughout SOLON. They are program objects which have 
among their attributes (slots) a list ot LISP functions (usually only one) to be evaluated when 
the state-operator has been activated. These functions can, in a multiple-processor 
("parallel machine") environment, easily be converted into individual processes running on their 
own CPUs. In this way the SOLON system is evolvabie in two important respects: 

First, new state-operators may be added and integrated into tine overall 
state-operator network structure without disruption to other parts of the net, as it is 
deemed necessary to subdivide jobs among state-operators or handle new dimensions of the 
planning problem; 

Second, state-operators which are currently functions or processes sharing a single 
CPU resource may be moved off to other processor machines as the work-load grows to an 
point where multiple processors become important for maintaining required real-time 
performance. 
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s"\. Abstract j 

Planning is a aa ce— ry bask for intelligent. adaptive systems 
operating independently of human controller*. Tiw — p*p»r 

rspcu*— «n j mi*ion planning system that performs task 
planning by""" decomposing i high-level miisnon objective into 
subtasks and ryntnesizrog a plan for those tails at varying 
levels of abstraction- We use a blackboard architecture to 
partition the search space \nd direct the focuj of attenuon of 
the planner. Using advahqed planning techniques, tvs. can 
control plan synthesis for the complex planning osis m- 
-voived m mission planning. \ j I 

. J i iS 
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.{jjbiAsO- 


{ y^i 


Planning is a necessary task for intelligent, adaptive systems 
operating independently of h man controllers. Autonomous 
systems need to plan their -croons and adapt themselves to 
erv penmen ui changes for survival. Given a high level nms- 
s:cn specification. the mission planning module needs to syn- 
thesize a sequence of actions to achieve mission goals. This 
.-ecu: res advanced techniques that reason about constraints, 
granularity of search, spatial configurations, levels of abstrac- 
:n and temporal ordering*. Robotic systems benefit from 
these planning techniques by me raising their independence 
from mission control. .As a rasu.t, expanded missions with 
reduced supervisory control can be conceived rod executed 1 . 


Various approaches are being used m planning systems. 
STRIPS uses means-ends analyse m robot pro clem icivtng by 
h ratifying differences from a state description to a goai rod 
selecting forward production ru.es to reduce them". NOAH 

-ses levels of abstraction rod a l east commitment strategy to 
generate parallel pans hierarchically for in mdoc* mobile 
ronct J . OPM uses opportunistic reasoning to xive the errand 
manning problem ry combining both tcp-ccwu rod rottom-up 
panning activities 4 MCL3HN uses skeletm refinement to 
pm experiments m mc*ecular rociogy by instantiating specific 
rperations from i sequence of gerermnec plan steps- 
PROTEAN uses i clack beard architecture to configure protein 


it. res ’y -ea.se mn g 


weh is the prcc.emracivmg process 3 . >ur mission planner 
bunds in these earner systems rod uses cev lsrwcts :*mm 


mmemented wironn the 


iron. lecture rod _ses the features 
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3. Mission Planning 

"be : veroV centre! of m tor emeus j’-rem requires the 

management :f mult; pie subsystems coot^ranng to achieve 
mission goals. One such subsystem J rotuxuia puinrung. "his 
paper reports on i mission planning system ouut for be 
tutnnoscus ope ration of FMCs autonomous .and v*mcie, ro 
Mi:3 tracked vehic.e. 


Misricn planning is the p r oce s s of syntheaizrog a sequence 
of actions to tttisfy goals and constraints poaed by the mis' 
«on manager. Mission plans are specified at varying levels 
of abstraction, wuh ausnon profiles at the higher levels and 
command sequences at the lower levels. "im.mand sequences 
are fixed to perform specific tasks given the 'chicle’s operat- 
ing characteristic* as such, they are best sequenced by com- 
puter programs that perform table lockup* Contention 
among these commands can usually be resolved at the higher 
levels, thus interaction among commands a minimal. At the 
other end of the planning hierarchy, mission profiles specify 
objectives and time tables for accomplishing the objective*. 
The vehicle achieves these objectives by executing command 
sequences downloaded by Mission Control at 'he appropriate 
times specified, in the mission profile- Mission profile* lend 
themselves to template or scr.pt planning because they are 
specified at a level of detail higher in the abKiacticn hierar- 
chy where mterfc.tion among the objectives is minimal. 

Tasks, on the other hand, ire synthesized nto plans by ocn- 
nderrng the currant state of the mission- Tasks consist of 
command sequence* an autonomous vehicle executes to achieve 
some part of the overall unseen. The planner performs task 
planning by decomposing the high-level mission objective into 
sub tasks and synthesizing i plan for those tasks it varying 
levels of abstraction. Intermediate tasks must be selected roc 
sequenced tn such a way that subsequent goa-s can be ach- 
ieved. .An exemplary ask performed by the planner is to 
develop a plan for conducting reconnaissance :n a particular 
ires specified by the mission commander. ~cr example. i 
mission to conduct ara* reconnaissance a necessary when the 
ccmmancer desires specific information about lertain locations 
or faculties within a defined area. To accomplish this mis- 
sion. the planner must find overwatch positions for raccn- 
ncitenng the urge ts, atabhsh routes sequencing these post- 
icus. rtta-n nea-th x the operation rod rapert ill mfer- 
maticn rapidly and accurately^. 

Tasks refer to the mtermediate abstraction levej :n the 
-.rorirg merara ny. tncae lsv»Ls where attraction among 
planning lecisicns j the highest. Interaction am eng tasks fi- 
ber by sequencing tasks with prerequisite rod postrequiscte 
oonditiocs or by decern posing tasks into ju busks makes up 
'hie mterasting planning problem. These interactions occur .a 
a dynam-caidy changing environment rod rraate a ocm- 
nnatenai expiates of the panning tpace: the *earoh through 
tha dynamic planning space s i key jsue for mssion pi.ro- 


4. Blackboard .Architecture 

U’e are implementing our mission panzer 
31ackxuro fvent Troves ivrem 5 . a v-rs:. 
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blackboard architecture 7 . As such. it defines probkm-*olvinf 
knowledge wurces for synthesizing plan steps, a muJc-levei 
solution blackboard for recording partial plans and a flexible 
control structure for con trolling the expansion of the planning 
space. 

Using a blackboard, a hierarchy of -tstnetion levels where 
each level represents a partial state description of the world 
at some time, we can partition the search space and direct 
the focus of attention of the planner. We map the problem 
space onto the blackboard by specifying abstraction levels in 
the plan hierarchy. These levels r epres en t both special and 
conceptual attractions for the mracoo planning problem, ^or 
the mAion of area rrermna usance, we generate a visibility 
map by creating boundary regions that contain locations 
visible to the target — a spatial abstraction. For the path 
planning task of the mission, we generate one type of oon- 
trafficable region by creating water boundaries — a con ccrt u al 
abstraction. Dtu attractions help control the exponential 
search process required in pLarming by establishing planning 
j Lands; areas whera Local search can find plan anchors for 
attaching the remainder of the plan. The more independent 
the planning sland*. the better the planner controls its plan- 
ning space by relying on Local search. The blackboard struc- 
tures the planning space in the proolem domain. To this 
structure, we apply the problem -solving strategy of skeletal 
plan refinement. 

When a miaion is specified, the planner chooses a general 
design. We specify a design with only the essential detail 

oeceamnr to direct the initial search of the plan. This Least- 
commitment strategy a maintained throughout the plan 
refinement process. The design specifies spatial configurations 
for plans and partitions the planning space into plan seg- 
ments. Once these segments are found, the planner succes- 

sively refines its plan by mstanuatiag plan steps at the 
lower levels. Plan instantiations occur by creating planning 
elements -ising the -orrect data abstraction with the current 
plan abstraction. At the design Level, the planner ca nn ot use 
low-level data to fora decisions. Instead, it uses high-level 
symbolic objects that repre s ent the relationships between the 
*asks that make up ‘he mission. 

For example, consider a plan for 1 reconnaissance mission 
that synthesizes a sequence of tasks in both tune and space 
such that the final configuration satisfies overriding objectives. 
A good design specifies the spatial orientation for each i the 
tasks. Finding this design depends on the reconnaissance tasks 
involved and their relationships to each other. At this Level 
of abstraction, the planner reasons about the target Location, 
the type of reconnaissance mission, visibility maps, non- 
t riff:. cable regions, military strategy and communication re- 
quirements. Only liter refinement :f ‘he design ran plan 
steps mvcivmg txact task Locations be mstanuatec ismg pixei 
data represented is coerdirste triples. 

At. Controlling ?!an Synthesis 

?Lan ryathesu occurs when kno w-.eoge source* instantiate 
pLan steps recorded m he blackboard hierarchy. Without 
controlling plan lyadesa. he planning system wculd ixnaus- 
tiveiy create he soiuncn space ox‘ possible plans. Whue his 
works for simple punning problems, m mission planning, is 
he complexity of he mission mcreases, he number of asks 
grows ind he numoer of potential plans grows exponentially. 
We use a three- tiered structure for ’.ary mg control over he 
execution of knowieege source* m he mission planning rye- 
tern that consists x eszebiishinf focus Jrdsicnj. executing 
strategies ind ~<srddj\f trusted ft scurces. During prociem 
scivusg, know Lee ft icurces create daemon elements m he 
plan hierarchy — is planning proceeds, mere knowledge curces 
ire Activated m a oeccme available for execution. A con- 


troller rates these knowledge sources using focus d eciaon s. 
strategies and rankings, and a scheduler selects a knowledge 
source to execute by choosing the ooe with the highest rating. 

Focus lecmooi re presen t collections of heuristics against 
which knowledge sources are rated 6 . These decisions establish 
criteria used to evaluate the utility of knowledge sources. 
Foe each knowledge source the controller calculates a utility 
value by summing together, for each focus daemon, the 
products of s focus weight r ep resent in g the value of a focus 
drciwsi and a w turf action level, the degree to which a 
knowledge nunx aatufio* a focus de ca M n . This calculation 
results in ratings that prioritize the knowledge sources so s 
scheduler can select the knowledge source with the highest 
rating. Focus denseons are crested during problem solving in 
response to changes in pUtming and reflect the general be- 
havior of the system. They add high-level control daemons 
that the controller uses to direct the generation of plan seeps. 

Strategies provide a rigid control structure that directly con- 
trols the execution of knowledge sources. They permit the 
execution of a strict mquence of knowledge sources. A 

strategy re p r e sen ts a procedure for achieving a particular goal 
and consists of a goat a status, a raoooaje and a list of 
strategies and tactics. The goal denotes what the strategy 

will tccom r lish when its status becomes operative, and the 
rationale describes what the strategy accomplishes. The ;r- 
dered List of strategies and tactics defines the specific subgos is 
that make up the procedure. When strategies are operative, 
knowledge sources that achieve the seme goals of the opera- 
tive strategies receive higher priorities than ones that achieve 
different goals. Focus decisions are used to differentiate be- 
tween knowledge sources with the same goau. 
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Figure -*-l: FINT)- LOCATION ind FIND-Rl strategies. 

F.gure -t-l illustrates the structure of two strategies usee ry 
the Nliraica Planning System- The first strategy. F ND- 
LCCATICN. consists of three tactics: INSTANTlATE- 

LOCATLON. RATE-LOCATION and CHXJSE-BEST-L0CA7LCN. 
Thu strategy finds a Location by creating -*r*nc** of ja- 
uccs, rating them md choosing the wt me. The seccco 
strategy, FIND-Rl. xnsuts :i the tactic LNSTA.VTTATE- 
LOCAL- AREA- R 1 and the strategy FIND- LOCATION. Thu 

recursive definition faoiitate* creating sew smxtegies from :x- 
utmg me*. Thu strategy finds a rositicn for performing 
racc nr -u- s a m ce cy creatmg mstur.ee* ; . jcu. irru men Laz- 
ing Locations within tb#** areas. 

The third Level of control m this three- red structure. 
ranking knowledge sources , overlaps with 'he preceeding two. 
Ranking pnon tires knowledge sources that are grcupec 
together because of similarities m fuacmc n or strategy. 
During system design, knowledge sources are ranked to in- 
ferential* between subtleties n their performance characaru- 
maa Usually, performance refer* to procewmg speed with 
procesmg speed determining the granuor.ty if search. Uma- 
.ng give* he controller a discriminating factor when 
chooses imeng know.roge scurces with he same ratmg. 
Thus, ranking hscr.m. nates between enow. sage sources 'hat 


would otherwise be considered equaL Knowledge «jun» 
with the special rank of IMMEDIATE b y pam the controller 
and execute immediately. 


5, Constraint- baaed lMmiif 

Another technique for coot roll: jg «*rch m \mr tg aao«rxmt» 
to Imut the number jf acceptable plan* (Xr planner u am 
constraints based cn terrain feature information. resource 
limitations, vehicle Limitation* and military doctrine. In th» 
way. the space of pdble plaamof solution* a con* rained 
by the specifications f the muon requirement*. A »*ai 
must meet certain jboctivee while satisfying coo* rain a that 
limit the success i an operation. The harder* the coo- 
stramta. the Leas flexible the plan and the earner t ta to con- 
fine the search. Aa constraint* become softer** they ooo- 
tnbute leas to coofininf the space of pomible plana. <>ir 
muon planner uses hard constraints to limit the number f 
acceptable plana by reasoning about terrain feature infor- 
mation. resource limi tation*, vehicle limitations and military 
doctrine during the planning proceaa. 

M_»on planning a data iruensive; therefore. *arch must be 
performed using different Levels of granularity. use a 

strategy that satisfies hard constraints before considering the 
«yt constraint* Failure to sausty any cf the hard con- 
straint* results .n the planner either ternunating it* search >r 
backtracking by coemdenng new potential solution* Our 
planner perform* simple ncktracking by expanding its search 
through the data baa* 3 y continually increasing the resolu- 
tion of the search, the planner increase* the number A data 
points that it cocmdm during planning. This technique al- 
lows it to male uniform cuts through 'die planning space is 
;t refines plans :op-dowm through the plan hierarchy. The 
planner performs acre complicated backtracking by modifying 
constraint* thereby achieving the obptuve but with some Lam 
A opumaiiry. The planner relaxes constraints by propagating 
hard constrain t* la our example of a recon naimance muon, 
an original ccnstraint j to use the rrmnnarsaance technique of 
tr-angulation. However, when this ccnstraint cannot be 
satisfied. the system relaxes it .a to one that allows straight 
reccnnaisBLnce. The ccnstraint remains a hard constraint; .1 
must be saf filed to complete the mission, but a plan allow- 
.rg for straight -eccnr.aiaance 3 Leae degrade than x-* that 
uses tnangulation. 

Our planner went* first from hard constraints to find a 
solution and backtracks only when necessary. It uses soft 
constraint* but considers them with Leas pnonty. Using con- 
strained search, we confine the planning space to cne that 
satisfies the hard constraints then find a miution that 
satisfies most if the soft constraint* As an examp.* con- 
sider the problem to place different sized rejects m 1 Leather 
touch. A effectively simple strategy places the Larger terns 
.a first, then squeezes :n the cru.Ier me* This itrategv 
works w*U because the planning space 3 characterized cv the 
flexibility :f the Leather pouch. \Ls*ca planning has 1 
smuiar flexible planning space because miauocs ire defined to 
•rtcompasa changing .one: tic r3 nat iccur .;m s he :cerst:cn 
f 1 Ticaoon. Thu* we use 1 strategy that sat^fie* the hard 
ccttstrxmts. much Like placing the Larger items m '-be Leather 
pouch first, then squeezes he soft constraint* mto place. 


■ Hard ccnstrxma -efer *0 thoae constraints that must 
satisfied when finding 1 vxLd solution. 

*■ 5cft xnstraina 'tier to those xnstnmti that niv r ruv 
het -jc a usf Led when finding 1 v*jd scuuticn. 


6. Mission .-leaning Imlta 

Tim Nlaaoa Planning System is written m Zeta. ap on a 
Sym bot es Lap Machine and mtrrfacaa to (fiber software 
modules asedad to control FVCt autonomous Land vehicle. 
The planning system oooata of IDS domain knowledge 
mu roe* t control knowledge sou roe* 27 strategies and |7 tac- 


Ftgure 6-1 Aowi the planning atate of the Minina Plan 
aing System in the intermediate stages of finding 1 plan for 
conducting raconnanaanoa In this example, the planner ooo- 
struct* a plan to moonnoiter an area using tnangulauoo tech 
irpite tha. gather information about the target. It sym- 
~ hi tea a plan by tiara ting knowledge scairosa and posting 
information m the blackboard. The blackboard Levels are 
shown from Use Strategy level down to the Rout* ievet 
Strategy. Tact** and Fan make up the Control Blackboard, 
while Muon. Dmgn. Region. Script. Lot. Location and Route 
make ip the Domain Blackboard. \ slues are depicted m ooe 
A four state* underlined values re p re s e n t operative entena 
that the planner cocnderi when making decisions; value* 
ihown .n reverse- video denote the current activated nibgoais 
n the planning space; boxed values ate the selected position* 
and routes that make up the final plan; and sh a de d values 
see figure 6-2) depict goals that have been accomplished. 



Figure 6 - 1 ; Muon Planning State in Cycle 21 
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Figure 6-2: Muon Planning State m Cycle 32 

After receiving a muon statement frem the commander f 
the *ehjci* the piamung system begins *0 find a poibie 
..an. It starts by ;reaung focus iecaurc* that provide top- 
down xmtroL This behavior change* is he rvstem jperatm 

ind rreatea xher xntroi decision* A ie«ge a *iected baaec 
:a the requirement for tr^nguuuon that deatifiee 'be taaaj 
teceaaary to acccmpLsh the mi»oo. Tie* taakj are confined 
to a f ffwn that 3 computed based :n vehicle and terraca 
constraint* A'lth the -egtea computed, the start and jea 
Locacccs are posted for reference by the planner. A icnpe 
-eauLtmg n a -ecccnajeance mission isos ( tnangulation s 
selected ana a strait f? for mstaohaung aenpt s 

i'enen^d. LHere. -he planner rapiemena zc Rl Z 2 strategy. 



Mtaf trtofmtodon Ik flnl plan to lyr to a* aotoa at 
d» Uotooi tto toatt towto. la thto Vaa plan. the whlrto 
ti awto along BOUTE1 fm too « arrta g penteon a ths ftnt 


iwmIbm loatoac. Rl. Aftoe wawtorii (to ttf g u. it 
tawli along ROUTES m the —and r— — location, 
tl At (to point. th* whacto Trtonfla— tod wqmtimd 

by cacnrtsg id rti final 1—laarhw itoaf KXTEX 


7. Cm!— Ian 

We daw toult a If— na naaataf Symw capable of m- 
ipienrtig ta^a to achtow hig her level ra— na o b j Ktiw a. W« 
ton bailt thto a j— i — af a blackboard architecture that 
dtfuMB knowledge aarrx a maiti- level blackboard and a 
flexible coo trot stnactara. U— g thto architacture integrated 

with other planning Tnrhnirp— , wa haw pm degree of con- 
trol onr tha erpkmw —red space inherent in m— on plan- 
nag problems 
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arc of occupancy, by itself a very fuzzy image of the world. Sev- 
eral hundred readings together produce an image with a resolution 
often better than 15 centimeters, despise many aberrations in indi- 
vidual readings t Figure 3). The resiliency of the method has been 
demonstrated in successful multi-hour long runs of Denning robots 
around and around long trajectories, using ihiee second map build- 
ing and three second map matching pauses at key intersections to 
repeatedly correct their position. These runs work well n clutter, 
and survive disturbances such as people milling around the running 
robot 

Xen Stuart of MIT and Woods Hole mis .mpiememed a three 
dimensional version of the sonar mapper for use with small sub- 
mersible craft Tested so far only in simulation, but in the oresence 
of large simulated errors. Stuart’s program provides extremely t,ood 
reconstructions. :n a \ZS « 128 < 64 uTay. of large scale terrain, 
working with about 60. (X)0 readings from a sonar transducer with 
a seven degree beam. Running on a Sun computer, his program can 
process sonar data fast enough to keep jp with the approximately 
one second pulse rate of 'he transducers on the two candidate suo- 
mersibies at Woods Hole. 

Recently Serey and Matdues demonstrated :he utility of the gnd 
representation in a stereo vision oased navigator [!J. Edges cross- 
ing a particular scaniine in the two stereo .mages are matched by 
a c>ru*miv programming method. o prixt-cc i 'ir.ge cron’. \ The 
wedge shaped space from die camera to the range profile is marked 
empty, cells along the proiilc itseif ire maned occupied. The re- 
sulting map is then used to pian obstacle avoiding paths as with 
the stereo and sonar programs mentioned a Dove i Figure -th 

Despite its effectiveness, m each nstancr we adopted :he gnd 
representation of space reluctantly. This may reject harms r'rom 
a recent time when analytic approaches were mere feasible and 
seemed more elegant because computer memories were too small to 
easily handle numerical arrays of a few thousand to a nuL* on ceils. 
I dunk the reluctance _s x> longer aopropr.ate. The straightfor- 
wardness. generality and uniformity x the grid representation has 



rtgist 2: 1 WniOte Patfc Plmmmn - \ pteft u whosca Ate mmuiuzes a 
cost fitecuoa hi a Ccrumcv Gnd- Small pcnurtMUora trr made w the vertices 
yf Ac paiii a* Arectwn* Ate reduce Ac cose 


proven itself in hmte eiement approaches o problems in physics, 
n raster based approaches to computer graphics, and has the same 
promise in roboac spatial representations. At hrsx glance a grid’s h- 
nite resolution seems inherently to limit positioning accuracy. This 
impression is false. Cameras, sonar transducers, laser scanners and 
other long range sensors have intrinsic uncertainties and resolution 
Urn its that can be matched by gnds no larger man a few hundred 
ceils on a side, giving l few thousand cells in two dimensions, or x 
few million in three dimensions. Since the accuracy of most trans- 
ducers drops wrh range, even greater economy is possible by using 
a heiraichy of scales, covering the near held at high resolution, and 
ucceisi'-c.y argernrges *;th .rKTeasingiy : curse r nods. 3esicrs 
this, die implicit accuracy of a certainty gnd can be better than die 
size of its ceil. The gnd can be thought of as i discrex sampling 
of a continuous : uncoon. Exxnded features such as lines < perhaps 
representing wails \ may be located to high precision by examining 
the paramexrs of surfaces of best liL The Denning robot navigator 
mentioned above convolves rwo maps to rind the displacement and 
rotation between them. In the final stages of the matching correla- 
tion values are obtained for a number of positions and ingles in the 
vicinity of the oest march. A quadratic least squares polynomial :s 
ritxd to the correlanoo values, and us pea* is ocared analytically. 
Controiled tests of the procedure usually give awmons accurate to 
betxr than one quarter of a ceil width. 
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Figure 4: Stem Mapptef aad Navgatwa - ?ten vie* of the certainty gnd 
built by a stereo guided robot traversing oir laboratory. Tbe situation a anak> 
job to Ac sonar case of Ftgwc 3, bid Ac range probles we gathered frora t 
scaniuK stereo method using two TV c a m er as rxber Am a sonar nng. 


Our results to suggest that many mobile robot asks can 
be solved with this unified, sensor independent, approach to space 
modelling. The key ingredients are a robot centered, multi reso- 
lution, map of the robot’s surroundings, procedures for efficiently 
inserting data from sonar, stereo vision, proxunny and other sensors 
into the map. other procedures for updating the map to reflect the 
uncertainties introduced by ur. precise robot mocon, and yet oth- 
ers to extract conclusions from the maps. We're already demon- 
strated procedures that produce local and global navigational fixes 
and obstacle avoiding paths ftvm such maps. Other tasks, suer 
as tracking corridors, finding \ mage points with good views of 
unseen regions, and idennficaDon of larger fcarses such as doers 
and desks by general shape seem within reach. 


3 The Representation 

The sonar mappers mentioned above are our most thorough use to 
date of the certainty grid idea. Although our xtginal implemen- 
tations used two grids to represent occupancy icow ledge (labeilec 
? ^ JJ and Stuart’s 3D system uses omy one. An analysis 

of *he iters n our code reveals that one pr.d vill miced suffice, 
ind this simplification makes c ear several puzaung -ssues in me 
original formulation. 

Before any measurements are made. Ac zrd is imdaiized a 
i background occupancy certain ry value. Cb. "his number rep- 
resents Ac iverage occupancy certainty we tvxci in a mature 
map, and encodes a ivery) little bit of a-pr.on juormanon we hare 
about the world In our Lab i good Cb seems x> be about the 
number of ceils in die perimeter of the gnd uvided by the too. 
ceils 4 x 32. (32 x 32) = I/S) in the case of rte Denning cooc. 
If die space s very ci uttered. Cb should be arger. .As the mar 
is used, values near Cb will stand for regions those occupancy 
staie s essennaily unknown, * rule these mucr nearer zero v- 





represen t empty places, and those much nearer unity axe likely to 
be occ u pied. Most of the planning algorithms chat use the grid 
will be better off if they do not make sharp distinctions, but in- 
stead numerically combine the certainty values from various cells 
to produce "goodness of fit** numbers for their various hypotheses, 
la this way the essential uncertainties in the measurements are not 
masked, and the algorithms do not jump to unnecessary, possibly 
false, conclusions. 

4 Inserting Measurements 

The readings of almost any kind of sensor can be incorporated 
into a certainty grid, if they can be expressed in geometric terms. 
The information from a reading can be as minimal as a proximity 
detector's report that there is probably something in a certain region 
of space, or as detailed as a stereo depth profiler’s precise numbers 
on the countours of a surface. 

The first step, in general, is to express the sensor’s measurement 
is a numerical spaaal certainty distribution commensurate with the 
grid’s geometry. For an infrared proximity detector this may take 
the form set of numbers P t in an elliptical envelope with high 
certainty values in a central axis 'meaning detection is likely there) 
tapering to zero at the edges of the illumination envelope. Let’s 
suppose the sensor returns a binary indication that there is or is 
not something in its held of view. If the sensor reports a hit. ceils 
in the certainty grid C x falling under the sensor’s envelope can be 
updated with the formula 

C x t = C x ■+■ P t — C g x P z 

which will increase the C values. In this case ’die P values should 
be scaled so their sum is one, since die measurement describes a 
situation where there is something somewhere in the field of view, 
probably not everywhere. If the reliability of the sensor is less than 
perfect, the normalization may be :o a sum less than unity. If. on 
the otner hand, the detector registers no hit, the formula might be 

Cx := Cx < 1 - Pz) 

and the Cs wiil be reduced. In this case the measurement states 
mat there s nothing anywhere in me field of view, and the P values 
should reflect only the chance that an object has been overlooked at 
each particular position; i.e. they .mould not be .normalized. If the 
sensor returns a continuous value rather than a binary one. perhaps 
expressing some kind of rough range estimate, a mixed strategy' 
similar :o the one described below for sonar is called for. 

A Polaroid sonar measurement is a number giving the range of 
the nearest object -within an approximately thirty degree corse in 
front or the sonar transducer. Because of the wide angle, the co- 
ject position is known only ;o be somewhere cn a certain surface. 
This range surface can be handled in the same manner as the >en- 
5 icvity distribution of a proximity aetector ’*hit'* above. The ionar 
measurement has something rise to sav. however. The volume of 
■.he cone ip *o the range reccing s -rrrudy empty. r.re ; -m.a.Icr 
range *ould have been returned- The empty volume s iike the 
‘no hit'* proximity detector case, and can be handled in the >ame 
rashion. So a sonar reading is iike a proximity ce teeter hit at some 
ocarcns. and increases the occupancy prooaoiiity there, and fike 
a miss at others, wncre ;r decreases the prooaciiiry. If we have a 
large numeer of sonar readings taxen from different vantage points 
say as the robot moves), me gradual accumulation of <uch cer- 
tainty numoers wall build a respectaole map. Ae can. :n tact, do a 
little reiter than that. Imagine two <cnar readings whose volumes 
mtersect. And suppose the 'empty * region of me recond overlaps 
car. of the range surface of the first. Now *hc range rurrace uvs 


“somewhere along here there is an object", while the empty vol- 
ume says “there is no object here". The second reading can be 
used to reduce the uncertainty in the position of the object located 
by the firs* reading by decreasing the probability in the area of the 
overlap, and c o rr es pondingly increasing it in the rest of the range 
surface. This can be accomplished by reducing the range surface 
certainties R, with the formula R t R t x (1 - E t ) where E, is 
the “empty" certainty at each point from the second reading, then 
normalizing the Rs. This method is used to good effect in the ex- 
isting sonar navigation programs, with the elaboration that the £s 
of many reelings are first accumulated, and then used to condense 
the Rs of the same readings. (It is this two stage process that led 
us to use two grids in cxir original programs. In fact, the grid in 
which the £s are accumulated need merely be temporary working 
space.) 

The stereo method of Serey and Matthies provides a depth pro- 
file of visible surfaces. Although, like a sonar reading, it describes 
a volume of emptiness bounded by a surface whose distance has 
been measured, it differs by providing a high certainty that there 
is matter at each point along the range surface. The processing of 
the “empty" volume is the same, but the certainty reduction and 
normalization steps we apply :o sonar range surfaces arc thus net 
appropriate. The grid cells along a very right distribution around 
die range surface should simply be increased in value according 
to the “hit" formula. The magnitude and spread of die distribu- 
tion should vary according to the confidence of the stereo match 
at each point. The method used by Serey and Matthies matches 
edge crossing alon^ .orre spending scaniines of two images, and is 
likely to be accurate at those points. Elsewhere it interpolates, and 
the expected accuracy declines. 

If the robot has proximity or contact sensors, its own motion 
can contribute to a certainty grid. .Areas traversed by the robot are 
almost certainly empty, and their cells can be reduced by the “no 
hit" formula, applied over a confident sharp edged distribution in 
the shape of the robot. This approach becomes more -interesting if 
the robot’s mocon has inherent uncertainties and inaccuracies. If 
the certainty grid is maintained so it is accurate with respect to the 
robot's present position (so coiled robot co-ordinates i, then the past 
positions of the robot will be uncertain in this co-ordinate system. 
This can be expressed by jiumr.% the certainty grid accumulated 
from previous readings in a certain way after eacn move, to reflect 
the uncertainty in that move. New readings are inserted without 
clur essentially the robot is saying “I know ->.ur:/v where I am 
row; I’m just not sure where I was before). The rack in the 
cenainty gr.d of a moving robot’s path in this system will resemble 
the vapor trail of a high flying jet - right and dense n the vicinity of 
the robot, diffusing eventually to nothing with true and distance. 

5 Extracting Deductions 

The curpose of maintaining a crrrainty grid in ore rooot is to pi an 
ir.d monitor letters. Theme tre zlfes -howee :*re *ay *o plan 
restacie avoiding paths. Conceptually the gnd can be considerec 
an array of topographic values * high occupancy ;ertaint:es are 
hills *niie .ow certainties ore vadeys. A sale parii fodows valleys, 
uke running water. A relaxation agonthm can perturo ponions o; 
a tnai path to bring each pan to a local minimum. In pnnc.pie 
i decision reed never be made is to which ociuens ire a c ready 
empty' and which are occupied, though perhaps ire program should 
stop it the rest path dimes re>crd some threshold “altitude". If 
-he moots sensors continue :o crerate and create the end as the 
path :s executed, .mpasses will re-come obvious is proximity one 
contact e risers raise the xcuparcy certainty :f locations where 
trey max? rentact *uh <oiid miner. 



As indiciTrd in the introduction, we have already demonstrated 
effective navigation by convolving certainty grids of given loca- 
tions built at different times, allowing the robot to determine its 
location with respect to previously constructed maps. This tech- 
nique can be extended to subpam of maps, and may be suitable 
for recognizing particular landmarks and objects. For instance, 
we arc presently developing a wall tracker that fits a least squares 
line to points that are weighted by the product of the occupancy 
certainty value and a gaussian of the distance of the grid points 
from in a-priori guess of the wail location. The parameters of the 
least squares line are the found wall location, and serve, after be- 
ing transformed for robot modon. as the initial guess for the next 
iteration of the process. 

For tasks that would benefit from an opportunistic exploration 
of unknown terrain, the certain ry grid can be examined to find 
interesting places to go next. Unknown regions are those whose 
certainty values are near the background certainty Cb. By applying 
an operator that computes a function such as 

V ( c. - Cb ) 1 

over a weighted window of suitable size, a program can find re- 
gions whose contents are relatively unknown, and head for them. 
Other operators similar in spirit can measure other properties of 
the space and the robot's state of knowledge about il Hard edged 
charactenzadons of the stuff in the space can be left :o the iast 
possible moment by this approach, or avoided altogether. 

6 A Plan: Awareness for a Robot 

Uranus is the CMU Mobile Robot Lab’s latest ana best robot and 
•he third and last one we intend to construct for the forseeable 
future. About 60 cm square, with an omnidirectional dnve sys- 
tem intended primarily for indoor work, Uranus carr.cs two racks 
wued for the industry' standard VME com outer bus. and can be 
upgraded with off the shelf processors, memory and muut output 
boards. In the last few years the speed and memory available on 
single beards has begun to match that available in our mainframe 
computers. This removes the mam arguments for operating the 



"igure 5: The Uranus Mobile - A V-mimg Mty. u.l :( itefnae. 


machine primarily by remote control With most computing done 
on board by dedicated processors, enabling very high bandwidth 
and relaible connection of processors to sensors and effectors, real 
time control is much raster. .Also favoring this change in approach 
is a realization by us. growing from our experience with robot con- 
trol programs from the very complex to the relatively snnple, that 
the most complicated programs are probably not the most effective 
way to learn about programming robots. Very complex programs 
are slow, limiting the numebr of experiments possible in any given 
time, and they involve too many simultaneous variables, whose 
effects can be hard to separate. A manageable intermediate com- 
plexity seems likely to get us to our long term goals fastest. The 
most exciting element in our current plans is a realization that cer- 
tainty grids are a powerful and efficient unifying solution for sensor 
fusion, modon planning. landmark identification, ind many other 
central problems. 

As the core of the robot and die research we will prepare a 
kind of operating system based on die ^certainty grid” idea. Soft- 
ware running continuously on processors onboard Uranus will 
maintain a probabilistic, geometric map of ±e robot’s sur- 
roundings as it moves. The certainty grid representation will allow 
this map to be incrementally updated in a uniform v,ay from var- 
ious sources including sonar, stereo vision, proximity and contact 
^nsors. The approach can correctiv model die fuzziness of each 
reading, while at the same tune combining multiple measurements 
to produce sharper map features, and it can deal correctly with un- 
certainties in the robot’s mocon. The map will be used by planning 
programs to choose clear paths, identify locations by correlating 
maps), identify well known and insufficiently sensed terrain, and 
perhaps identify objects by shape. To obtain both adequate res- 
olution of nearby areas and sufficient coverage for longer range 
planning, without excessive cost, a hierarchy of maps will be kept. 
;he smallest covering a 2 meter area at 6.2 5 cm resolution, die 
largest 16 meters it 50 cm re sol u con (Figure 6). This map viil 
be “ scrolled” to keep die rooct eerie red as it moves, but rotations 
of the robot wii] be handled by changing elements of a matrix the 
represents die robot’s orientation in die grid. The map forms a 
kina of consciousness of the world surrounding the rooot - reason- 
ing about the world would actually oe done by computations :n the 
map. It might be interesting to take one more step in the hierar- 
chy, to a one meter grid that simply covers the robot’s own extent. 
It would be natural to keep this final grid oriented with respect 
to robot chassis itself, rather than approximately to the compass as 
with die other grids. This c range of co-ordinate system veuid pro- 
vide a natural disuncuon between "world” awareness ana "body" 
or “seif” awareness. Such encoding of a sense of seif mi gr.t even 
be useful if the robot were covered with many sensors, cr pemaps 
were equipped with manipulators. A'e have no immediate nans in 
hat direction, and so wail pass by his interesting dea for new. 

Our initial version wail rontain a pair of two timenr.or.ai end 
set s, one mapping me presence of oojects ui me roocts roeranng 
■eight cr" a few feet aboe mound ’eve!. The other will mao me ess 
complex :cea or presence of passac.c door a: various .councils. The 
object map will be updated mom ail sensors, the fioor mao pnmar- 
ily from downward locking proximity detectors, meugn possibly 
also from long range data from vision and sonar. The robot wail 
navigate by dead reckoning, integrating he mcr.cn ?r 3 v reels. 
This method accumulates error rapid v. and his uncertainty be 
retieexd m he maps by a repeated biurr.ng operation. Old reac- 
mgs. *nose location relative :o me robots ore sent rosinon and 
orientation are known with cecreasing precision, will rave me: r 
effect gradually df fused ?v this pe ration, until mey eventually 
evaporate :o me background certxnty value. 
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Figure 6: Map Resolution Heirarcby - Coarse macs far ibe big pKnire. line 
ones for ihe iddly :n ihe jnmediate environment. Ail the maps are 

scrolled to keep the robot in ’he center cells. 

It would be natural :o extend the two-grid system to many grids, 
each mapping a particular vertical slice, until we have a true three 
dimensional grid. We 'will do this as our research results, and 
processing power permit The availability of single board array 

processors that can be installed on the robot would help this, as 
the certainty gnd operations are very amenable to vectorizing. The 
certainty gnd representation can also be extended in the time di- 
mension. with past certainty grids being saved at regular intervals, 
like frames in a movie dim. and registered to the robot’s cunent 
co-ordinates (and blurred for motion uncertainties). Line operators 
applied across :ne time dimension could detect and track moving 
objects, and give the react a sense of time as well as space. This 
has some very mnlling conceptual ; and perceptual') consequences, 
but we may not get to it for a while. 

Even die simplest versions of die idea wall allows us fairly 
straightforwardly to program the robot for tasks that have hitherto 
been out of reacn. We look forward to a program mat can explore a 
region and return to its starting piace, using map ‘snapshots" from 
its outbound journey to and its way back, even in the presence of 
disturbances of is mccon and occasional manges in the terrain. 
3y funneiing the sensor readings through a certainty grid, which 
collects and preserves ail die essential data, and indications of un- 
c trainees, and makes .t avadable in a uniform way, we avoid the 
rroolem we've cad. that for each combination of sensor and task a 
afferent program is required. Now the as* execution is decoupled 
from the sensing, and mus becomes simpler. 
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1. .ABSTRAQO? / 

; f? r-* * ' ■ *■ ~' r> 

WiyaSRSWR architecture for object modeling and recognition for an auto- 
nomous land vehicle/^ Examples of objects of interest include terrain features, 
fields, roads, horizon features, trees, etc. The architecture is organized around a 
set of data bases for generic object models and perceptual structures, temporary 
memory for the instantiation of object and relational hypotheses, and a long term 
memory for storing stable hypotheses that are affixed to the terrain representa- 
tion. Multiple inference processes operate over these databases. We^Iescribe 
these particular components: the perceptual structure database, the grouping 

processes that operate over this, schemas, and the long term terrain database. 

rnBirillfr".“ith a processing example that matches predictions from the long 
term terrain model to imagery', extracts significant perceptual structures for con- 
sideration as potential landmarks, and extracts a relational structure to update 
the long term terrain databases ^ 


A 


u. . 


2. INTRODUCTION 


Terrain ana object models for autonomous land vehicles (A LvV are 
required for a wide range of applications including route and tactical planning, 
location verification through the recognition of terrain features and oojecrs. anc 
acquiring new information about the environment as it is explored. The follow- 
ing lists important criteria for terrain and object modeling capabilities. 


Descriptive Adequacy: The modeling technique should be capable o: 
describing *.ne objects and situations In the environment necessary for the vehicle 
to function. This Induces representing natural as well as man-made objects. It 
should be a consistent representation that supports modular system development 
and uniform inference procedures that can operate over diferent types of objects 
at diferent levels of detail. Uniform shape, object subpart ana surface attribute 
affixments are necessary *o :o this. 


Recognition Adequacy: Much of the activity of an .VLV Is :or:cernec wit. 
determining where it is and what is around :t. Terrain models snouid he mam- 
p i.ac.e .or ueter::t;;..ng trie itnscr-cujea ippear-Uc rs ;; vcr.u ^ tweets n;t c 
controlling recognition processing. This invokes the formation of ;enera. predic- 
tions of sensor derived features from the terrain mocel. iuen predictions wil. 
often be uncertain and qualitative due to incomplete pr:o* knoweage of tne ter- 
rain. 


Handing Uncertainty: The existence ana exact environmental ocation x 
objects will often not oe enow - with complete certainty. Locations will often a? 
determined relative to otner known locations me not with respect to a giooahy 


313 


consistent terrain map. This is true, for instance, when the sensor displacement 
parameters are not well determined. It is necessary to represent this uncertainty 
explicitly in the terrain model so incrementally acquired information can be used 
for disambiguation. 

Learning: A vehicle will learn about the environment as it moves through 
it. Associating new information wuth the terrain representation should be 
straightforward. This is difficult to do. for example, by changing values in a raw 
elevation array. Types of information to be affixed to the terrain representation 
include newly discovered objects, details of expected objects, and the processing 
used in object recognition. 

F usion of Information: The ALV must build a consistent environmental 
model over time from different sensors. As an object is approached, its image 
appearance and scale will change considerably, yet it has to be recognized as the 
same object, with newly acquired information associated with the unique instance 
of the general object type. In a typical situation, a distant dark terrain patch 
will be partially recognized based upon distinctive visual characteristics, but may 
be either a building or a road segment. As it is approached, its image appearance 
changes considerably, making disambiguation possible. This requires the 
representation of multiple hypotheses, each formated with respect to the proper- 
ties of the potential world objects. The structure of the object description should 
direct the accumulation of information. 

A further consideration in developing and evaluating terrain modeling capa- 
bilities is that there is not a single ALV. Instead, there are a wide range of auto- 
nomous vehicles, indexed by a diverse range of active and passive sensors md 
assumptions about a priori data. There is a continuum from systems having a 
complete initial model of the terrain and perfect sensors to those with no a priori 
mode!, and highly imperfect sensors. For example, a robot with no a priori data 
and only an unstabilized optical sensor will probably model the environment in 
terms of a sequence of views related by landmarks and distinct visual events 
embedded in a representation that is more topological than metric. .An ALV 
solely dependent on optical imagery will have to deal with the huge variability in 
the appearance of objects. Experience has shown that even road surfaces have 
nighty variable visual characteristics. Alternatively, a few pieces of highly pre- 
selected visual Information can serve to verify predictions from a reliable and 
detailed terrain model and precise position and range sensors. 

We call a general object model a schema. A schema can represent per- 
ceived. but unrecognized, visual events, as %veil as recognized objects and their 
-eiationships in environmental scenes. The architectural design is focused about 
the representation, instantiation, and inference over schemas developed by the 
ALV is it moves through the environment. Schemas are related to similar con- 
cepts found in Hanson et.aL - 7$ and Ohta - SO . The short term terrain 
-ep-esentatlon tons is" s of schema instantiations that mo resent accumulated cer- 
ceptuai evidence for objects as attributes and relations that are hypothesized 
with varying levels of certainty. 

Ooject models are used to organize perceptual processing bv integrating 
descriptive representations with recognition and segmentation control. One 
aspect of this is the use of different types of attributes and inheritance relations 
between generic schemas for -epresentaiion in !S-A and PART-OF hierarchies. A 
particular object attribute relates three dimensional world properties of an object 
inc sensor dependent view information, either by a set of generic views or 


viewing procedures. These viewing attributes are also inherited and modified 
according to different object types. In many systems, objects are treated as lists 
of attributes that are matched against extracted image features. Here they are 
treated as specifying an active control process that directs image segmentation by 
specifying grouping procedures to extract and organize image structures. 

Another critical aspect of the architecture is the various types of spatial, 
localization relations that deal with uncertainty and learning by associating 
different types of perceptually derived information with terrain models. For 
example. local (multi-sensor) viewframes affix sets of schemas and un-recognized 
perceptual structures into local “ robot *s-eve* % views of an ALV’s environment. 
Path-affixments between local viewframes support fusion of information in time 
without necessarily corresponding to locations in an a priori grid. 

This effort has developed an architecture for terrain and object recognition 
compatible with the wide range of potential sensor configurations and the 
different qualities of a priori data. 

There has been work in artificial Intelligence, computer vision, and graphics 
that satisfy the individual requirements for object modeling capabilities, but little 
has been done to integrate them. To date, there is no vision system that can 
interpret general natural scenes, although some can deal with restricted environ- 
ments Hanson et.al. - 7$ while other systems are restricted to artificial objects 
and environments. Brooks* Brooks - 54 representation based on generalized 
cylinders meets, or could be extended to deal with, many of these functions. It 
has well defined shape attribute inheritance between a set of progressively more 
complex object models, and affLxment relations that could be generalized to han- 
dle uncertainty, [t can also be used to generate constraints on image features 
from object models. Nonetheless, the system built around this representation has 
had limited success beyond dealing with essentially orthographic views of 
geometrically well defined man-made objects. This appears to be partially 
because the constraints on image structures generated from the abstract instances 
of object models are too general to generate initial correspondences between 
models and image structures. Brook’s system also used an impoverished set of 
image descriptions, and the object models could not direct the segmentation pro- 
cess directly during their instantiation. The majority of work in terrain modeling 
deals with how well a representation can realistically model three dimensional 
terrain, but not how It : s used for recognition. The simplicity of a mocei that is 
described by a few parameters is not useful for recognition unless it can direct 
constrained searches against Image data. For example. Pentland*s Pentland - S3 
use of fractals satisfies aspects of descriptive adequacy for natural terrain, but has 
been ies- effective '“or recognition. Kuipers Kuipers - 52 has produced an 
interest*!; terrain model for learning and handling uncertainty, but it Is non 
visual. Rt ated to this Is Kuan s Kuan - 54 object based terrain representation 
for planning that : s organized in *erms of distinct, modifiable objects, but is also 
not assoc, a tea wit a sensor aerivea processing resuits. 


3. ARCHITECTURE OVERVIEW 


The system architecture ‘onsists of 
processes. The inference processes transform 
data structures, and modifying * he existing 


several databases and inference 
the databases, creating additional 
ones. The task interface focuses 
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attention in system processing and monitors progress toward system task goals. 
This high level architecture is depicted in Figure 1. The boxes with square 
corners in this figure represent databases, the ellipses represent inference 
processes, and arrows indicate dataflow. 


3.1 SYSTEM DATABASES 

At the highest level there are three databases. These are the short term 
memory (STM), long term memory (LTM), and generic models. 

The STM acts as a dynamic scratchpad for the vision system. It has two 
sub-areas, a perceptual structures database (PSDB) and a hypothesis space. The 
PSDB includes incoming imagery from sensors, immediate results of extracting 
image structures such as curves, regions and surfaces, spatial temporal groupings 
of these structures, and results of inferring 3D information. 

The hypotheses space contains statements about objects and terrain in the 
world. A hypothesis is represented as an instantiated schema. The schema 
points to the various perceptual structures in the PSDB that provide evidence 
that the object represented by the schema (such as a terrain patch, road, tree, 
etc.) exists in the world. Other types of hypotheses include grids, viewframes, 
and viewpaths. Grids are a special type of terrain representation that contain 
elevation information and are derived from range data or successive depth maps 
from motion stereo. Viewpaths, as partially ordered sequences of viewframes. 
give space time relationships between hypotheses. Viewframes are sets of 
hypotheses that correspond to what can be seen from a localized position. A 
hypothesis with no associated perceptual structures is a prediction. As structures 
and localization are incrementally added to a hypothesis, it progresses on the con- 
tinuum from predicted to recognized. Hypotheses that have enough evidence 
associated with them to be considered recognized and stable, are moved to the 
LTM. 

4 
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Figure 1: Terrain Modeling and Recognition System Architecture 
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The LTM stores a priori terrain representations, the long term terrain data- 
base, and hypotheses with enough associated evidence to be considered visually 
stable. A priori data concerning elevation and terrain type information, as well 
as knowledge of specific landmarks are stored in the LTM. A viewframe. 
representing a certain location in the world is stored in the LTM if the evidence 
associated with it could be re-used to recognize the local environment if it was 
re-encountered. Consistency of one hypothesis with another is not required for 
storage in the LTM. 

The model space stores generic object models, the inheritance relations of 
the (model) schema network, and a set of image structure grouping processes and 
rules for evaluating image structure interestingness. Generic models are used 
dynamically to instantiate and guide search processes to associate evidence to an 
object instance. Inheritance relations are used by various schema inference pro- 
cedures to propagate structures, attributes and relations between object instan- 
tiations. For instance, the generic two-lane-road schema has an “IS- A” relation- 
ship to the generic road schema. It follows, based on the inheritance models, that 
an instantiation of the two-lane-road schema will inherit the more general charac- 
teristics of the generic road schema that in turn inherits the more general charac- 
teristics of a terrain patch. Unlike the STM and LTM, the model space is not 
modified by inference processes. 


3.2 INFERENCE PROCESSES 

At the highest level, there are five different sorts of inference processes in 
the vision system. These are perceptual inference, location inference, object 
instantiation, LTM STM instantiation, and the task interface. 

The PSDB is initialized with the output of standard multi-resolution image 
processing operations for smoothing, edge extraction, flow field determination, 
etc. Much subtler inference is required for grouping processes that produce con- 
nected curves, textures, surfaces, and temporal matches between image struc- 
tures. These grouping operations are typically model guided. There are generic 
models which may be task dependent) of what constitutes “interestingness** of 
an image structure. 

The hypothesis inference processes produce tasks for the perceptual 
processes. These may be satisfied by simple queries over the PSDB such as “find 
ail long lines in this region of image*', where “long", “line" and “region** are suit- 
ably interpreted. Queries can be more complex, requiring, for instance, temporal 
stability, such as “find ^il homogeneous green texture regions that are matched 
ii.e.. remain in the field of view) over at least two seconds of imagery'**, where, 
again, qualitative descriptors are rigorously defined. Alternatively, the requested 
perceptual structures may be dynamically extracted. In this case, a history of the 
processing attempts ar.c results are maintained. Lf similar requests are mute 
later, such as if we were to view the same environment from a different perspec- 
tive. these processing histories could be used to recall a processing sequence that 
produced successful results. 

Location processes include a number of different modes of spatial location 
representation and inference. While exact location information is used when it is 
available, a key concept is the qualitative representation of relative location. 
This is fundamental, because the problem of acquiring terrain knowledge from 
moving sensors involves handling perceptual information that arises from 
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multiple coordinate systems that are transforming in time. The basic approach 
to location inference is to represent the location of world objects in a qualitative 
manner that does not require the full knowledge of continuous transformations of 
sensor coordinates relative to the vehicle the sensors are mounted on. or of 
transformations of vehicle coordinates relative to the terrain. 

The main structures involved in location inference are viewframes. 
viewpaths, and grids. Viewframes represent both metric location information 
about world objects derived from range sensors and view-based location informa- 
tion about the directions in which objects are found derived from passive sensor 
data. 


Generic schemas are models of world objects that include information and 
procedures on how to predict and match the object models in the available sensor 
data. Besides representing 3D geometric constraints, 2D-3D sensor view appear- 
ance including effects of change in resolution and environmental effects such as 
season, weather, etc., schemas also indicate contextual relationships with other 
objects, type and spatial constraints, similarity and conflict relations, spatial 
localization, and appearance in viewframes. 

Object schema instantiation may occur by model-driven prediction from a 
priori knowledge, or directly from another instantiation and a PART-OF relation. 
The other instantiation process may also occur by matching a distinctive percep- 
tual structure to a schema appearance instance. This sort of “triggering’’ is more 
common in situations where there is little a priori information to guide predic- 
tion. Object instantiations generate queries to the PSDB grouping process in 
order to complete matching. 

A key idea in object instantiation processing is inference over the model 
schema network hierarchies. Direct representation and inference over a large 
enough body of world objects to accomplish outdoor terrain understanding 
requires very large memory and proportionately lengthy inference procedures over 
that memory' space. Hierarchical representation makes a significant reduction in 
storage requirements: furthermore, it lends itself naturally to matching schema to 
world objects at multiple levels of abstraction, thus speeding the inference pro- 
cess. Two basic hierarchies are the IS-A and PART-OF trees. 

IS-A hierarcnies represent the refinement of object classification. Figure 2 
shows part of an IS-A hierarchy for terrain representation. The level of 
terrestrial-object tells us that we will not see evidence of any schema instance 
below this node as perceptual structures surrounded by sky. At the level of 
terrain-patch we pick up the geometric knowledge of adherence to the ground 
plane, while information stored at the level of a road schema constrains the boun- 
daries of a terrain patch to be locailv linear (with other constraints). Types 
beneath road add critical appearance constraints in color and texture, while the 
final refinement level In the IS-A hierarchy, the number of lanes, further ton- 
strains size parameters inherited from the road schema. 

PART-OF hierarchies represent the decomposition of world objects into 
components, each of which is. itself, another world object. Figure 3 shows a 
PART-OF hierarchy decomposition for a generic 2-lane-roaa. PART-OF hierar- 
chies contain relative geometric information that is useful in prediction and 
search. 
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Figure 2: IS- A Hierarchy 


Figure 3: Part of Hierarchy 


As object instantiation inference reasons up and down schema network 
hierarchies, incrementally matching perceptual structures and other data to 
instances of object appearance in the world, a history mechanism records the 
inference processing steps, parameters and results. This dynamic data structure 
is called the schema instantiation structure. One important aspect of this struc- 
ture is that it can used to extract the inference and processing sequence(s) that 
worked earlier to see the same object, or ones that are similar. This accounts for 
the fact that distinctiveness in image appearance is an idiosyncratic process that 
depends upon many factors which are difficult to model and control, such as 
current motion, wind, varying outdoor illumination, etc. 


4. PERCEPTUAL PROCESSING AND THE PSDB 


Perceptual processing is concerned with organizing images into meaningful 
chunks. The definition of •‘meaningful” and the development of explicit criteria 
to evaluate segmentation techniques involves, from a data-driven perspective, 
that the chunks have characterizing properties, such as regularity, connectedness, 
and not tending to fragment the image. From a model-driven point of view, seg- 
mentation appropriateness corresponds to the extent to which the pieces can be 
matched to structures and predictions derived from object models. T rom either 
perspective, a basic requirement is that image segmentation procedures find 
significant image structures, independent of world semantics, in order to initialize 
and cue model matching. This allows for the extraction of world events such as 
surfaces, boundaries, and : nteresting oatterns ^dependent of understanding per- 
ceptions in the context of a particular object. These, in turn, are useful abstrac- 
tions from image information to match against object models or describe the 
characteristics of novel objects. 

The Perceptual Structure Data Base (PSDB), conceptualized in Figure 4, 
contains several different types of information. These are classified as images, 
perceptual objects, and groups. Images are the arrays of numbers obtained from 
the different sensors and the results of low level image processing (such is con- 
tour extraction and region growing routines) that produce such arrays. It s 
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difficult for the symbolic relational representations used for object models, such 
as schemas, and the processing rules in computer vision systems, to work directly 
with an array of numbers. Therefore, there are many spatially-tagged, symbolic 
representations used in image understanding systems that describe extracted 
image structures such as the primal sketch _Marr - 82 , the RSV structure of the 
VISIONS system Hanson et.al. - 78 , and the patchery data structure of Ohta 
Ohta - 80,. We found it useful to build such a representation around a set of 
basic perceptual objects corresponding to points curves, regions, surfaces, and 
volumes. 

Groupings are recursively defined to be a related set of such objects. The 
relation may be exactly determined, as in representing which edges are directly 
adjacent to a region, or they may require a grouping procedure to determine the 
set of objects that satisfy the relationship. Groupings can occur over space, e.g., 
linking texture elements under some shape criteria such as compactness and den- 
sity, or over time, as in associating instances of perceptual structures in succes- 
sive images. We stretch the concept a bit, so that groupings also refer to general 
non-image registered perceptual information, such as histograms. 


4.1 INITIALIZATION OF THE PSDB 

Whenever new sensor data is obtained, a default set of operations are per- 
formed to initialize the PSDB. Edges are extracted at multiple spatial frequen- 
cies and decomposed into linear subsegments. The edges are extracted into dis- 
tinct connected curves, and general attributes such as average intensity, contrast, 
and variance are associated with them. Similar processing is performed for 
regions extractions. Histograms are computed with respect to a wide range of 
object based and image based characteristics in a pyramid like structure. These 
default operations are used to initialize bottom-up grouping processes and schema 
instantiations. These, in turn, determine significant structures using heuristic 
interestingness rules to prioritize the structures for the application of grouping 
processes or object instantiations. 


4.2 IMAGES 

Images are the data arrays derived from the optical and laser range sensors 
and the results of image processing routines for operations including histogram- 
based segmentation, different edge operators, optic Sow field computations, and 
so forth. .Associated with images are several attributes for time of acquisition, 
relevant sensor parameters, etc. Processing history is maintained in the process- 
ing relationship structure that keeps track of the processing history of ail objects 
in the PSDB. 


4.3 PERCEPTUAL OBJECTS 

Points, curves, regions, surfaces, and volumes are basic types of perceptual 
structures that are accessible to object instantiations and grouping processes. .Am 
example instance of a curve structure is shown in Figure 5. This figure shows 
many common representational characteristics of perceptual objects. There are 
default attributes associated with particular objects, such as endpoints, length 
and positions for a curve. There is aiso an associated attribute-list mechanism 
for incorporating more general properties with an object. This list is accessible 




Figure 4: Perceptual Structure Data Base (PSDB) Figure 5: Curve Example 
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Figure 4: Parallel Grouping 


by keywords acd a general query mechanism using methods specific to the partic- 
ular associated attribute. The associated attributes in the example are shown in 
capital letters. There are many types of attributes that can be consistently asso- 
ciated with a curve using this mechanism. 

A useful representation for performing geometric operations and queries 
over objects is the OBJECT LABEL-GRID (or GRID: in the example curve. 
The number 6 indicates the index of this structure). This is an image where each 
pixel contains a vector of pointers back to the set of perceptual objects and 
groups which occupy that position. This allows geometric operations to be per- 
formed directly on the grid. Filtering operations can be applied to the OBJECT 
LABEL GRID to restrict processing based upon attributes associated with 
objects. Various types of masks can be associated with objects to reflect a direc- 
tional or uniform neighborhood to determine object relationships in the OBJECT 
LABEL GRID. 
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4.4 GROUPS 


A group is a set of related perceptual objects. The relation can be deter- 
mined directly by a query over an object and those surrounding it, as in finding 
the set of curves within some distance of a given region. Alternatively, it may 
require a search process to find the set of objects meeting some, potentially com- 
plex, criteria. For example, an ordered set of curves can be grouped together 
using thresholds on allowable changes in the average contrast and orientation of 
successive elements. By expressing the grouping process as a search over a state 
space of potential groups, each group becomes a potential hypothesis in the 
PSDB. Groups can also reflect temporal relationships; this occurs in matching 
structures in successive images. A relational grouping procedure is shown in Fig- 
ure 6 for the determination of nearby parallel lines with opposite contrast direc- 
tions. This is done for a linear segment by first extracting nearby neighbors 
using a narrow mask oriented perpendicular from the segment at its mid-point. 
The intersection of this mask with points in the label grid are determined, and 
then each candidate is evaluated by checking if it is within allowable thresholds 
for length, contrast, and orientation. It is then ordered with respect to the smal- 
lest magnitude of the difference vector computed from the average gradients. 
The grouping processes can either produce the best candidate as a potential 
grouping, or some set of them. 

Two different types of grouping processes have been developed: measure- 
based and interestingness-based. The measure based grouper is a generalization 
of established edge and region linkers Martelli - 76;. It uses a measure consisting 
of: 


.) some value to be optimized, such as length, minimal curvature, com- 
pactness, or a composite scalar value 

2) local constraints on allowable changes in attributes 

3) global thresholds on attributes 


The measure and associated constraints are optimized by a best first search 
returning several ordered candidate groups. The measure to be used can be asso- 
ciated with a prediction from an object model for substance or shape characteris- 
tics. The measure to be optimized can also be determined directly from initially 
extracted objects by selecting those that are extreme in some attribute or are 
correlated with the attributes of surrounding objects to derive a measure to be 
optimized. 

The measure based grouper is currently being generalized into one based on 
interestingness, it involves the basic processing .oop shown in Figure 7. initially, 
basic perceptual objects including curves, regions, junctions and their associated 
attributes are extracted using conventional techniques. Extracted objects are 
represented in label grids to express spatial neighborhood operations over the 
objects. A uniform neighborhood is established for each object, and directed rela- 
tions are formed with the adjacent objects in each neighborhood. These relations 
are represented in a small number of types of match relationships that contain 
descriptions of the correlation of attributes, subcomponent matching, and compo- 
site properties. 
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Selected attributes of the extracted perceptual objects and the match struc- 
tures are then sorted into lists with pointers back to the associated objects. 
These lists are for attributes such as size, average feature values, variance of 
feature values, compactness, the extent of correlation between the components 
and attributes of different structures, and the number of groups an object is 
involved in. These different rankings are then combined using a selection criteria 
to choose the set of interesting perceptual objects and relationships. The selec- 
tion criteria sets the required position in different subsets of the sorted attribute 
lists. An example is to find 100 largest objects in the top 10 of any of the attri- 
bute correlation lists. The selection criteria is modifiable during processing and is 
meant to reflect the influence of model-based predictions. 

Interestingness is used to focus the application of grouping rules to a 
selected set of objects and relations between objects indicated in match struc- 
tures. The grouping rules then combine perceptual objects to form new percep- 
tual objects, or groups, based upon the type of relation between the objects. 
Neighborhoods are established with respect to these derived groups to form new 
relationships. These in turn are sorted in the attribute lists with respect to the 
previously extracted perceptual objects. In addition to the relations established 
in uniform neighborhoods, for some groups, non-uniform relations are also esta- 
blished. Processing can continue indefinitely as less and less interesting relations 
become candidates for the application of grouping rules. Explicit criteria are 
needed to stop processing; e.g., we can limit processing time, determine when 
there is a uniform covering of the image with extracted groups, or when struc- 
tures belong to unique groups. 
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Figure 7: Grouping Processing Flow Figure $: Grouping Architecture 
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These operations are performed by virtual processors called grouping nodes. 
Grouping nodes are seen as covering regular and adjacent portions of an image 
area (not necessarily of a single image, because there can be multiple images in a 
motion sequence). The image area contains some portion of a label plane for 
accessing the objects based upon their spatial dispositions as well as object-based 
associated attributes. The grouping nodes are further organized in a hierarchical 
pyramid shown in Figure 8. Each node is connected to its adjacent neighbors 
and has a parent and descendants. The transfer of information between nodes at 
different levels is based upon interestingness. Lower level processes send their 
most interesting structures up the hierarchy. There are several effects of this. 
One is that it allows a uniform processing to occur at different levels, so grouping 
rules can be applied to objects at different levels of interestingness. It also allows 
relations between nonspatially adjacent structures to be handled in a uniform 
architecture. It also partitions perceptual structures in a way that corresponds to 
different levels of control in instantiation of object models. 

Organizing segmentation in terms of grouping processes has many advan- 
tages for a model based vision system. The grouping processes can be run 
automatically from extracted significant structures based upon perceptually 
significant, though non-semantic criteria. Thus, connected curves of slowly 
changing orientation or compact, homogeneous regions can be extracted purely 
on perceptual criteria. These image structures correspond to world structure and 
events, and they are useful for initializing schema instantiations. They 
correspond to the qualitative image predictions associated with more general 
schemas. An inference process for compilation from an object model into group- 
ing processes, allows model based vision to have a very active character quite 
different from single-level attribute matching. 


5. SCHEMAS 


Schemas represent hypotheses about objects in the world. The process of 
schema instantiation creates an instance of a schema together with evidence for 
that schema. Evidence consists of structures in the PSDB, a priori knowledge 
stored in the LTM, predictions derived from location inference, and relations to 
already instantiated schema. 

Table 1 shows the various slots and relationships in a generic schema. 
Although this data structure has a frame-like appearance, it is useful to view the 
schema as a semantic net structure, with slots representing nodes in the net and 
relationships representing arcs. Schema instantiation inference reasons from a 
(partially) instantiated node, follows arcs, and infers procedures to execute from 
the sum of its acquired information in order to obtain more evidence to iurther 
instantiate the schema. 

The schema network is a generic set of data structures that indicate the a 
priori relationships between schemas. A key part of this network is the inheri- 
tance hierarchies that indicate which descriptions and relationships can be inher- 
ited from schema to schema. Inheritance hierarchies allow efficient matching of 
objects in the worid against sensor evidence from progressively coarser to finer 
levels. As reasoning moves from coarser to finer levels of description in model- 
based schema instantiations, the schemas inherit descriptive bounds and add new 
descriptions, and also add constraints to inherited ones. For example, the system 
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Table 1: Generic Schema Data Structure 
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may first recognize an object as a terrain patch (because it lies on the ground 
plane). A road is a type of terrain patch (see Figure 1, that adds linear boundary 
description, and constrains the visual image appearance of the terrain patch 
schema in the color and texture descriptors. The two basic types of schema net- 
work inheritance hierarchies are IS-A and PART-OF. 

Below is a brief explanation of each of the slots and relationships in the 
generic schema data structure. Schema type refers to the generic name of the 
schema in the IS-A hierarchy. Schema name is the identification of the schema 
instance, e.g., if the schema type is ‘‘road” then the schema name might be 
‘highway 101’. The schema instantiation structure maintains the control history 
of the schema recognition inference processes for this schema. 

The 3D description is an object-centered view of the world object 
•epresented by the schema. It includes its 3D geometry and shape description, 
actual size, and inherent color and texture (as opposed to how its color and tex- 
ture might appear to a particular 3ensor). Note that this is the description that 
matches the 3chema-object before looking at its structure refined into com- 
ponents. For example, the 3D geometric description of a tree schema does not 
separate the canopy from the trunk, but gives a single enclosing volume as its 
representation. The volumetric descriptions of the trunk and canopy appear as 
the 3D descriptors on their schema further down the PART-OF hierarchy. Thus, 
inferring down the PART-OF hierarchy corresponds to increasing the resolution 


325 


of the view of the object represented by the schemas. 

The sensor views are descriptions of the stable or frequently occurring 
appearances of the schema object in imagery. This description is intended to be 
used for image appearance prediction, evidence accrual for instance recognition, 
3D shape inference, and location inference. The reason for storing or runtime 
generation of explicit (parametrized) image views is that the perceptual evidence 
matches to these descriptions, not to the three dimensional ones. 

The distinctive image appearance slot holds descriptions of perceptual 
structures that are likely to occur bottom-up in the PSDB. They provide coarse 
triggers for instantiating the schema object hypothesis without prediction. 

The perceptual structure is the dynamically created PSDB query history 
generated by the schema instantiation as it attempts to fill in evidence matching 
the various schema slots and relations. The instantiator can re-use successful 
branches of perceptual structures to improve its recognition speed as it continues 
to view other instances of the same generic schema type. 

Components are pointers to other schema that represent sub-parts of the 
schema object. They are finer resolution description of the schema, one level 
down on the PART-OF hierarchy. The MUST-HAVE components are assumed 
to be parts the represented object must have to exist, although the schema may 
be instantiated without observing them all. Occasionally occurring components, 
such as center-lines on roads, can be stored in the NlAY-HAVE slot. Spatial rela- 
tionships between components as they make up the schema object are listed at 
this level also. Relationships can also be stored on a view dependent basis. 
These relationships access the sensor-view dependent data in that slot. PART- 
OF's point upward one level on the PART-OF hierarchy, indicating that this 
schema is a component of another schema. 

Classification points upward and downward one level on the IS-A hierarchy. 
There may be more than one such pointer, which Is to say that the IS-A hierar- 
chy may be partially ordered. 

Contextual relationships indicate spatial/temporai consonance or disconso- 
nance between groups of schema types, omitting those which are already indi- 
cated in the PART-OF and IS-A hierarchies. Schema that .ALWAYS or never- 
occur with the given one can be used strongly for belief or dis-beiief in the 
schema instance and as focus of attention mechanisms within the instantiation 
process. SOMETIMES occurs with relationships that are used to store the 
spatial-temporal aspects of schemas relative appearance in the viewed environ- 
ment. 


CONFUSED- WITH and SIMILAR-TO relationships indicate schema that 
may be mistaken for the given one, but for different reasons. One schema may 
ae confused with another oecause they share common evidence pieces, cut for 
which there are sufficient descriptors to disambiguate. Two schema are similar if 
there is sufficient ambiguity in their appearances, and therefore the available per- 
ceptual evidence, that they may be indistinguishable without contextual reason- 
ing. For example, tall grass may be confused with wheat from coarse shape me 
texture evidence, but can often be disambiguated by color descriptors or finer 
resolution examination of structure (because of wheat berries, for example). How- 
ever. roads are similar to runways because they cannot necessarily oe dis- 
tinguished by their Intrinsic appearance, no matter now detailed or accurate the 


descriptors and evidence. Contextual reasoning, e.g M the presence of aircraft oa 
the runway, global curvature of the road, etc. is required. 

Locational information points at the various viewframes the schema appears 
in and inferred 3D relationships with other world objects. 

Recognition strategies are prioritization cues for the schema instantiation 
processes that suggest inference chains likely to pay off to match this schema 
instance against sensor evidence. 

The recognition strategies slot in the schema data structure prioritizes infer- 
ence approaches relevant to this schema. These approaches include search for 
components, search for part of schema instance, search on weaker classification, 
relations with other schema instances, and PSDB matching. 

Search for COMPONENTS and search for PART-OF are both inferences 
along the PART-OF hierarchy in different directions. The instantiator searches 
the relevant slot to see if there are components to search for or another object of 
which this schema is a component. If the COMPONENT or PART-OF schemas 
exist, they can be accessed to continue the inference. Otherwise, each causes an 
instantiation of the missing schema to be generated as a prediction. Instantiation 
control can be transferred at this point to the COMPONENT or PART-OF 
schema. The schema inference process maintains its thread of reasoning relevant 
to the schema in the schema instantiation structure slot. 


8. LONG TERM TERRAIN DATABASE 


The long term terrain database is part of LTM. It stores the data neces- 
sary for a mobile robot to perform vision-based navigation and guidance, predict 
visral events, such as landmarks and horizon lines, and to update and refine 
maps. 


The long term terrain database contains a priori map data including 
government terrain grids, elevation data, and schemas representing instances of 
stable visual events recorded wnile traversing paths in the environment. The use 
of a priori map and grid data :o predict percepts and to heip guide image seg- 
mentation is shown in Section 5. The following presents a summary of a struc- 
ture for spatial representation and inference that enables a robot to navigate and 
guide itself through the environment. 

We first define the notion of a geographic ''place" in terms of data about 
visible landmarks. A place, as a point on the surface of the ground, is defined by 
the landmarks and spatial relationships between landmarks that can be observed 
:Yom a fixed location. More genera fiy, a place can be defined as a region n ipace. 
in which a fixed set of landmarks can be observed from anywhere in the region, 
and relationships between them do not change in some appropriate qualitative 
sense. Data about places is stored in structures called viewframes, boundaries 
and orientation regions. 

Viewframes provide a definition of place in terms of relative angles and 
angular error between landmarxs. and very coarse estimates of the absolute range 
of the landmarks from our point of observation. Viewframes allow the system to 
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localize its position in space relative to observable local landmark coordinate sys- 
tems. In performing a viewframe localization, observed or inferred data about 
the approximate range to landmarks can be used. Errors in ranging and relative 
angular separation between landmarks are smoothly accounted for. A priori map 
data can also be incorporated. A viewframe is pictured in Figure 20. 

A viewframe encodes the observable landmark information in a stationary 
panorama. That is, we assume that the sensor platform is stationary long 
enough for the sensor tr pan up to 360 degrees, to tilt up to 90 degrees (or to use 
an omni-directional sensor Cao et.al. - 86 ), to recognize landmarks in its field of 
view, or to buffer imagery and recognize landmarks while in motion. 

A sensor-centered spherical coordinate system is established. It fixes an 
orientation in azimuth and elevation, and takes the direction opposite the current 
heading as the zero degree axis. Then two landmarks in front of the vehicle, 
relative to the heading, will have an azimuth separation of less than 180 degrees. 
If we assume that no two distinguished landmark points have the same elevation 
coordinates (i.e.. no two distinguished points appear one directly above the other) 
then a well-ordering of the landmarks in the azimuth direction can be generated. 
We can speak of the landmarks as being “ordered from left to right”. The rela- 
tive solid angle between two distinguished landmark points is now well defined. 

Under the above assumptions, the system can pan from left to right, recog- 
nizing landmarks. L t , and storing the solid angles between landmarks in order, 
denoting the angle between the i-th and j-th landmarks by Ang t; . The basic 
viewframe data are these two ordered lists, ( L V L <>,...) and (Ang l2 ,Ang 2 3 ,...). The 
relative angular displacement between any two landmarks can be computed from 
this basic list. In Levitt et.al. - 87- we show how to use this data to essentially 
parametrize all possible triangulations of our location relative to a set of simul- 
taneously visible landmarks. This localizes the robot’s position in space relative 
to a local landmark coordinate system. 

View-frames contain two basic dimensions of data: the relative angles 
between landmarks, and the estimated range (intervals) to the landmarks. If we 
drop the range information, we are left with purely topological data. That is, it 
is impossible, using only the relative angles between landmarks, and no range, 
map or other metric data, to determine the relative angles between triples of 
landmarks, or to construct parametric representations of our location with 
respect to the landmarks. Nonetheless, there is topological localization informa- 
tion present in the ordinal sequence of landmarks: there is a sense in which we 
can compute differences between geographic regions, and observe which region we 
are in. 

The basic concept is to note that if we draw a lice between two (poL.t) 
landmarks, and project that line onto the (possibly not fiat) surface of the 
g-ound, then this line iivides the earth into two distinct regions. If we ran 
observe the landmarks, we can observe which side of this line we are on. The 
■‘virtual boundary" created by associating two observable landmarks together 
thus divides space over the region in which both landmarks are visible. We call 
these landmark-pair-boundaries (LPB's), and denote the LPB constructed from 
the landmarks L x and L 2 by LPB(I V L 2 1. 

Roughly speaking, if we observe that landmark L x is on our left hand, and 
landmark L 2 is on our right, and the angle from L x to L 2 (left to right) is less 
than 180 degrees, then we denote this side of, or equivalently, this orientation of. 
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the LPB by L t L 2 ]. If we stand oq the other side of the boundary, LPB(L V L 2 ), 
“facing” the boundary, then L 2 will be on our left hand and L j on our right and 
the angle between them less than 180 degrees, and we can denote this orientation 
or side as L jj (left to right). 

More rigorously, define: 

orientation-of-LPB(£ lt L 2 ) 

~ l if ®12 < * 

= sign(x-0 I2 ) = 0 if e, 2 = t 

-i if e l2 > x 

where 0 l2 is the relative azimuth angle between L x and L 2 measured in an arbi- 
trary sensor-centered coordinate system. Here, an orientation of -1 corresponds 
to the L i L zl side of LPB (L P L 2 ), -1 corresponds to the [L Z L X j side of 
LPB(L l ,i ; ) and 0 corresponds to being on LPB(L X ,L z ). It is a straightforward 
to show that this definition of LPB orientation does not depend on the choice of 
sensor-centered coordinate system. 

LPB's give rise to a topological division of the ground surface into observ- 
able regions of localization, called orientation regions. Crossing boundaries 
between orientation regions leads to a qualitative sense of path planning based on 
perceptual information. The three levels of spatial representation given by map 
or metric data, viewframes and orientation regions are pictured in Figure 9. A 




Figure 9: Muitipie-Leve’s-of-Spatial Representation 


natural environmental representation based on viewframes recorded while follow- 
ing a path is given by two lists, one list of the ordered sequence of viewframes 
collected on the path, and another of the set of landmarks observed on the path. 
We call the viewframe list a viewpath. The landmark list acts as an index into 
the viewpath, each landmark pointing at the observations of itself in the 
viewframes. For efficiency, the landmark list can be formed as a database that 
can be accessed based on spatial and/or visual proximity. Visual proximity can 
be observed, or computed from an underlying elevation grid and a model of sen- 
sor and vision system resolution. 

The first occurrence of a landmark points at the instantiated schema or per- 
ceptual structure in the vision system database that was used to gather evidence 
in the landmark recognition process. After that, ail recognized re-occurrences of 
this landmark point back at this initial instance. The same is true for the first 
occurrences and successful re-recognition of LPB’s and viewframes. This mechan- 
ism allows multiple visual path representations, built at different times, to be 
incrementally integrated together as they are acquired by using a common land- 
mark indexing pointer list. 

We use an environmental representation for orientation-region reasoning 
that is a list of oriented LPB’s encountered and crossed in the course of following 
a path. We call such a list an orientation-path. .As with viewpaths, there is an 
associated landmark list that indexes into the orientation-path. 

A dynamically acquirable environmental representation that merges the 
representations for viewpaths and orientation-paths consists of an ordered list 
interspersing viewframes. LPB crossings, and appearance and occlusion (or loss of 
resolution) of landmarks, as well as recording the headings taken in the course of 
following the path over which the environmental map is being built. Thus, we 
can integrate the representations required for viewframe and orientation region 
based reasoning with heading and landmark information to formulate an environ- 
mental representation that supports hybrid strategies for navigation and gui- 
dance. The representation is formed at runtime and consists of multiple inter- 
locking lists of sequential, time ordered, lists of visual events that include those 
necessarv for the navigation 3nd guidance algorithms presented in Levitt et.al. - 
87 . 


7. PROCESSING EXAMPLE 


The following processing example demonstrates the behavior of some imple- 
mented system components. These include the format of predictions from the 
long term terrain model, the extraction of perceptually significant groupings from 
“re PrfDB. how* an *nstant : atec schema uses grouping t> recesses and queries over 
the PSDB. ana extracting relevant cues for maxing viewframe localizations :n tne 
long term terrain representation. 

Figure 10 shows the elevation contours and road .twork in 'he a prior: ter- 
rain data from the Martin Marietta ALV 'est site in Denver which was supplied 
by the LIS. Army Engineer Topographic Laboratories (ETLL The vehicle posi- 
tion on the road is indicated by the arrow in the figure. From this, we are able 
to roughly determine the correspondence between an image taken from the road 
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figure 12: Predicted Segmentation From Grid Data 


and the terrain data, (the relevant sensor parameters were not available). Figure 
11 shows the terrain and feature classification supplied with the a priori data. 
These correspond to sets of image overlays in register with the elevation data. 
The road network is stored as a set of curve objects that is decomposed into 
linear segments with supplied attributes, such as road material and width. Ter- 
rain Batches are extracted as regions from terrain type information and 
parametric surface fits to the a priori elevation data. 

Figure 12 shows how the grid registered terrain data is instantiated into 
STM f o 'orm a predicted segmentation. The grid data regions from connected 
analysis correspond to schema instances in the Long Term terrain memory. Esta- 
blished surface display techniques are used to project the elevation with the asso- 
ciated schema instances to form a predicted view. Image positions are then 
labeled with their associated schema instances. Additionally, there are many 
schema instances, ordered by depth, at the corresponding image locations. The 
resulting predicted segmentation is processed as an abstract image where critical 
perceptual events are determined by size, adjacencies across occlusion boundaries, 
or types cf terrain with high semantic contrast, such as water, fields, or man- 
made structures. The perceptual structures are merged together based upon 
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distances and semantic type to yield predictions at different resolutions. 

Figure 13 shows the predicted terrain patches for the vehicle positioned 
with respect to the terrain in Figure 10. Figure 14 shows the predicted segmenta- 
tion after filtering to pull out the horizon line and road terrain discontinuities for 
roads near the vehicle. This data is quite coarse (30m sampling), and image areas 
in the foreground are highly composite containing instances of road and the adja- 
cent grassy fields. Nonetheless, the predicted segmentation yields a qualitative 
description of predicted image features that is sufficient to initialize and direct 
grouping processes to find corresponding image features and relationships. The 
key characteristics of the predicted segmentation are that the vehicle is on a flat 
plane, and that its field of view consists of road and grassy field terrain patches 
with some mountains in the distance. Predictions cf the dirt road off to the right 
and the intersection are made from the road-network and the elevation informa- 
tion stored along with it. The predictions are in terms of constraints on region 
adjacencies across boundaries, and the shape and attributes, such as color con- 
trasts, of the boundaries themselves. The horizon line constraints are that it will 
tend to have smoothly changing orientation and be adjacent to a large homogene- 
ous region (the sky). In general, the predicted features are described with con- 
strained attributes determined from the visibility components of schemas. 

Figures 15 and 16 show some of the contour related structures in the initial- 
ized PSDB. Figure 15 shows the edges extracted at one spatial resolution using 
the Canny edge operator Canny - S3 . We have found it useful not to apply 
noise suppression to extracted segments in order to base filtering on structural 
properties of the contours, including linear deviation and relationships to other 
image structures. Different linear segment fits for this extracted edge images are 
shown in Figure 16. 

Figure 17 shows the results of grouping processes applied to a set cf 
selected curves in Figure 12 with multiple associated attributes for orientation 
and color contrasts. The grouping processes were constrained by the predicted 
segmentation in Figure 14 using constraints on allowable color contrasts, changes 
in linear segment orientation, and rough image position and extent. Multiple 
groups are obtained for each predicted image event. Selection of one, or main- 
taining multiple alternative groups, is explicitly represented in the schema instan- 
tiation structure. Here, groups were selected based upon length and uniformity 
of composite attributes. 

F’gure 18 shows the results of a road schema instantiation based upon 
matches to extracted road boundar es in accounting for road surface properties 
through PART-OF relations. Texture elements adjacent to the road boundary 
which are consistent with a road surface, such as low contrast, parallel edges 
corresponding to ‘read marks, are used to direct queries to instantiate potential 
road area. Queries are also used to determine the presence of anomalous struc- 
tures : n the road such as anything which is high contrast or oriented perpendicu- 
lar to the road uirection. Sued structures require aisamoiguation through instan- 
tiation of another schema lit could be a road marking) cued by the anomaly or 
elevation estimates derived from motion displacements or range sensing. 

Significant image structures near the horizon line are particularly important 
for landmark extraction. Figure 19 shows extracted interesting perceptual groups 
near and above the horizon line. Figure 20 shows an extracted view-frame 
representing the relative visuai spatial relationships between some of the objects 
extracted from this field of view. 









Figure 18: Road Schema Instantiation Figure 19: Significant Perceptual Groups 
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Figure 20: VIewframe Instance 
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8. SUMMARY 


The architecture we have developed, using terrain and road schemas with 
implemented system components for . perceptual processing and manipulating long 
term terrain data, has been successfully used in tasks for ALV navigation and 
scene interpretation. 
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A. Introduction 

The goal of building robots with increasing amounts of autonomy from direct 
human control or supervision is a long term goal of the Telerobot project Among 
the primary missions for such robots are tasks in the areas of on-orbit satellite 
servicing, inspection, and various assembly operations associated with the space 
station. A research and development program with a series of demonstrations of 
increasing robot autonomy has been planned. This paper discusses some of the 
technical, methodological, and logistical issues of producing the artificial 
intelligence capabilities required by the long-term (1993 through 2000-era) 
demonstrations of these robot systems. A substantial research effort must begin 
now if the demonstration objectives are to be met on schedule. It is no coincidence 
that many of the same objectives for Telerobot artificial intelligence research are 
common to both the Telerobot and System Autonomy programs. Many of the 
capabilities and associated research required for autonomy transcend the particular 
application. 

The planned series of Telerobot demonstrations provides a relatively clear 
progression of increasing sophistication in the artificial intelligence capabilities 
required. As a directed research and development program, these demonstrations 
provide considerable applications pull in the kind of research issues which may be 
addressed. However, applications pull alone can easily become overly restrictive to 
new research ideas and also unrealistic about existing or projected technological 
capability. Basic research, on the other hand, provides the technology push towards 
new ideas. Ultimately, this push results in demonstrations and application of more 
powerful capabilities. Technology push by itself runs the risk of being distractable 
and undirected towards economically realistic and desirable application goals. 
Clearly, a mixture of both applications pull and technology push is necessary. 
Achieving a sensible balance between these two opposing forces will be major 
criteria of on-going and overall success of the Telerobot project 

Due to the scope of the artificial intelligence technology required by Telerobot, one 
of the conclusions to draw from this document is that our goals for Telerobot 
artificial intelligence will be difficult to achieve unless we make maximal use of all 
available research resources. NASA probably does not have the desire or capability 
to fund all of the artificial intelligence research which will be required by 
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Telerobot, nor should NASA duplicate research conducted under other auspices. 

We will need to draw extensively upon research results generated by the Systems 
Autonomy Program at the NASA Ames Research Center laboratory, and externally 
funded laboratories at major universities and within the AI industry. 

B. THE LONG-RANGE GOAL: CAPABILITIES 

This section details the specific capabilities which will be required in the area of 
Artificial Intelligence for the long-term Telerobot demonstrations leading to 
operational 2000-era autonomous robots. These capabilities should be regarded as 
necessary for the specific objectives of the demonstration program. However, they 
should not be construed to be a sufficient set of capabilities. This can only be 
established by continuing research, experimentation, and the demonstrations 
themselves. Subsequent sections will provide detail on some of the technical issues 
which must be addressed. 

B.l TASK PLANNING AND SCHEDULING 

A number of processes must operate at the Task Level to robustly control the 
achievement of goals. One of these is task planning and scheduling. Its function is 
to select and schedule activities whose collective effects logically achieve the 
desired goals. In the Telerobot, some of the planning and scheduling capabilities 
required ere: 

• Use of highly structured procedures. 

Many of the tasks which the robot will cany out have been engineered 
beforehand, e.g., there are satellite servicing procedures which must be followed 
in x definite, prescribed sequence. The robot should have the ability to select 
the appropriate procedure and combine it with other elements of its activity 
plan. It will be a rare case when predefined procedures can be simply "invoked” 
without regard to plan interactions or the potential for error due to real-world 
uncertainty. 

• Creation of ad hoc routines. 

The robot must be able to reason about the task requirements of novel goals. 

All situations in which the robot is expected to behave cannot be conceived or 
engineered beforehand. There will undoubtedly be circumstances when a 
structured procedure has not been developed. This will certainly be the case 
when a structured procedure fails and error recovery processes must create 
appropriate plans to handle the novel circumstances. The robot must be able to 
freely mix planning using structured procedures and ad hoc routines. 

• Management of task resources. 

In the accomplishment of any goal, the robot can draw only upon limited 

resources. There are a finite number of effectors available, a finite number of 
sensors to employ, a finite number of interchangeable or unique tools or 
replacement parts, etc. Management of resource constraints is a fundamental 
component of creating viable plans. 
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• Planning with uncertainty. 

The ability to formulate robust plans when there is considerable uncertainty 
about the run-time environment, the robot’s ability to affect it, or the knowledge 
used in planning (see Uncertainty management below). 

• Multi-agent cooperation. 

The ability to formulate plans which provide for specific tasks to be 
accomplished in a coordinated fashion by other intelligent agents, robot or 
human. Also, the ability to formulate plans which are robust in circumstances 
where the actions of other intelligent agents are not prescribed or are otherwise 
unpredictable. 

BJ2 SPATIAL PLANNING AND REASONING 

The function of spatial planning and reasoning processes is to bring the logical 
procedures of activity planning into the real-world by making geometric, physical, 
or temporal constraints on activities an integral part of the task level control 
process. This may require a theory of manipulative processes which support 
planning for handling, service, repair, construction, and inspection. Some of the 
capabilities which are required are: 

t Reasoning about robot, workspace, and workpiece geometry. 

Geometric constraints on tasks can play a significant role in the selection and 
sequencing of appropriate actions by the robot. Given a device to be 
disassembled for repair, the selection and sequencing of manipulations is 
dependent on the geometric configuration of the device, e.g., you can’t remove 
an orbital replacable unit (ORU) without removing an access paneL Contact and 
attachment constraints easily generate other examples where geometric 
reasoning is important. 

• Reasoning about physical processes. 

The ability to plan for or around the effects of physical processes is important 
When a tool is released in zero-g, the robot must understand how physical 
processes such as inertia will affect the state of the tool, e.g., will it float away 
and if so, where and at what speed? Often only qualitative answers are required 
in the planning process (e.g., the tool will rotate, float left, etc.), but the robot 
must be able to utilize precise quantitative information where necessary (e.g., to 
direct the movement of a manipulator towards a wrench which is floating away). 
Another example would be reasoning about the flow of fluid, such as fuel, 
through a hose in a refueling operation. The planning process must take into 
account physical constraints on planned activities. 

• Planning for workpiece and robot positioning. 

The robot must be able to reason about how to position itself and the workpiece 
to enable required operations, and when this is impossible, to provide the 
appropriate constraints for activity planning. In zero-G, without the default 

• constraints and orientation provided by Earth gravity, a mobile robot has 
considerable latitude in this area which should be exploited. 

• Planning for robot movement. 


341 



The ability to plan reasonable and safe manipulator trajectories through a 
cluttered workspace is a critical component of task level control. Collision 
avoidance with fixed obstacles, stationary but movable obstacles, and moving 
obstacles is a significant problem. 

• Mobility. 

The ability to plan the movement of the robot and navigate in real-time mrough 
a 3-dimensional environment The robot should have the ability to maintain its 
position relative to moving workspaces (Le., station keeping) as well as plan its 
movements so as not to conflict with other mobile objects (e.g., other robots, 
astronauts on EVA, or spacecraft). 

• Temporal Reasoning. 

Reasoning about the duration and timing of activities and physical processes 
introduces additional constraints on the task planning process. The scheduling of 
activities to accommodate real-world events, including the actions of other agents, 
will rely on an effective temporal reasoning ability. 

• World modelling. 

The ability to maintain a coherent geometric and physical model of the world as 
it dynamically changes as a result of robot actions, the actions of other agents, 
and controlled or uncontrolled physical processes is critical Virtually all task 
level control processes must have access to a (reasonably) consistent set of 
beliefs about the state of the world in order to make assumptions for planning, 
diagnosis, error recovery, and other functions. When conflicting information and 
ambiguity are introduced, this can become a serious problem. 


B.3 EXECUTION MONITORING 


In a sometimes malevolent and only partially- modelled real world, a robot 
executing a plan may not actually achieve the desired effects of each action while 
still obeying any constraints imposed by the plan. There are a host of potential 
problems which may create a divergerxe at execution time from the expected 
effects of actions as described in the plan. It is the role of execution monitoring 
processes to determine whether the plan is executing nominally, and to track the 
state of the world during plan execution. A number of capabilities are important: 

• Verification and sensor planning. 

The ability to determine how best to employ sensors during plan execution to 
verify that robot actions are achieving their intended effects, and to notice other 
events which may affect successful plan execution. Planning and scheduling, 
resource management, and sensor modelling techniques are aspects of this 
process. 


• Expectation generation. 

In order to determine that the intended eifects of a plan are actually occurring, 
qualitative and quantitative predictions about what information will be obtained 
on sensors during plan execution must be generated. 


• Situation assessment and sensor data fusion. 


In lieu of perfect models of the sensors and the world, the expectations about 
information on various sensors will be approximate at best Techniques must be 
developed to make accurate judgements of whether expectations are satisfied or 
violated from partial data in predictions and obtained from sensors. Frequently, 
information from multiple sensors will need to be coordinated in this process. 

^ Recognition of unexpected events. 

The ability to recognize and characterize events or states of the world which are 
unanticipated. In the formulation of plans, the robot will not have the ability to 
predict all events which may affect the successful execution of the plan. In 
order to robustly accommodate potential conflicts at plan execution time, the robot 
must first be able to notice and describe unexpected situations. 

B.4 DIAGNOSIS AND ERROR INTERPRETATION 

When errors or unexpected events are detected during plan execution, diagnosis 
and other error interpretation processes must attempt to discover the source of the 
problem. In the absence of a precise diagnosis, error recovery or restart procedures 
will be coarse and may needlessly cause the duplication cf work successfully 
completed or even the overall failure to achieve the required goals of the robot 
On the other hand, a precise diagnosis may indicate a simple procedure to recover 
(e.g., regrasp a tool if it slips during usage) or the need to plan a more substantial 
recovery (e.g., when the satellite's access panel is not attached as described in the 
CAD database). The following capabilities are required: 

• Localizing system failures. 

Identification of single and multiple point failures of the hardware of the robot 
or associated systems. This type of diagnosis is critical for determining the 
status of the robot and its ability to accomplish required tasks. 

• Knowledge-base criticism. 

Occasionally, the knowledge which the robot uses to plan actions, monitor, or 
otherwise solve problems will be missing or in error. The capability to spot 
these gaps or errors will be essential, especially when machine learning becomes 
an important process. 

• Debugging lower-level controllers. 

In some cases, lower-level controllers may operate incorrectly for a variety of 
reasons. The ability to make this decision (and possibly suggest a restart of the 
controller or other recovery action) is important. 

• Diagnostic test design. 

The robot should have the ability to devise and utlitize diagnostic tests to 
discriminate among multiple competing hypotheses. Some tests may involve 
passive inspection using sensors ie.g., use vision to determine if an access panel 
fastener is missing). Others might involve more active procedures which could 
compound the error if applied without due consideration (e.g., to check if the 
effector is jammed, rotating it in a confined area could damage nearby satellite 
components). 
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• Diagnostic techniques. 

The capability to employ a variety of different styles of diagnosis as required by 
the problem at hand Sometimes heuristic techniques will be applicable and 
serve to quickly localize a problem. In other cases, the ability to reason in depth 
about the structure and function of robot components will be important The 
ability to use causal modelling techniques to project the effects of possible 
failures will be useful in other circumstances. 

• Explanation. 

The ability to describe and justify diagnostic conclusions to the level of detail as 
required by a system operator ( see Human Interface below) or other knowledge 
based system. For effective error recovery, a diagnosis must have enough 
precision to specify the requirements for error recovery (e.g., "The forearm is 
broken" lacks sufficient detail for recovery). In addition, the ability to justify the 
particular conclusion reached (especially when there are closely competing 
alternative hypotheses) will be important for human evaluation and risk 
assessment of possible recovery procedures. 

B.5 ERROR RECOVERY PLANNING 

After a diagnosis has been determined, it is the role of error recovery procedures 
to plan how to get the plan back on track. This may involve small modifications to 
the plan at hand, the replanning of significant procedures, or even the 
abandonment of goals deemed impossible to accomplish in the changed 
circumstances. Some capabilities which will be important are: 

• Selection of predefined procedures. 

In some contingencies, recovery procedures will have been devised beforehand. 
The robot should have the ability to determine the applicability of these 
recovery procedures from a machine or human generated diagnosis. 

• Discrepancy analysis. 

The ability to determine what the effects of a particular failure are on planned 
actions, scheduled events, or actions in progress. In major failures, much of the 
plan remaining to execute will become invalid. However, in other cases the 
effects of failures may be slight and the existing plan resumed after some 
simple local error recovery. Discrepancy analysis will be critical in not wasting 
the work already put into planning and plan execution. 

• Ad hoc planning and plan integration. 

If predefined recovery procedures are inapplicable to the situation at hand, the 
robot must have the ability to specify the requirements of a novel procedure, 
invoke the appropriate planning process, and then incorporate the new procedure 
into :he existing plan. Precision in the ability to excise the invalid components 
of the old plan and splice in the new procedure will be probably depend on 
Discrepancy Analysis as described above. 

• Goal maintenance. 

The robot must be able to determine when a plan is irrecoverable and a goal 
must be abandoned or revised. In addition to the case wftere recovery is 


344 



7 * 


a : 


impossible, in some cases recovery may be too time consuming, too risky for the 
robot or workpiece, or unsafe for astronauts working nearby (e.g., venting fuel 
from a clogged or kinked refueling hose). 

• Revising and acquiring knowledge. 

If diagnosis processes have determined that some aspect of the robot’s 
knowledge was in error, misapplied, or missing, a recovery procedure must 
determine how to correct the problem. This is one aspect of the machine 
learning problem, resolution of which will certainly be important in the 
achievement of full robot autonomy. 

B.6 SIMULATION AND PREDICTION 

Central to much of the robot’s ability to solve problems is the capability to create 
and reason about the potential extended effects of alternative actions or events. In 
order to plan, the cumulative effects of alternative actions must be considered. In 
order to anticipate the long-term effects of potential failures, the immediate effects 
of the failure must be extended forward through time. Sometimes, in order to 
discover the cause of current anomalies, the ability to reason how the effects of 
hypothetical previous failures could propagate will be important 

• Causal Simulation. 

The ability to utilize causal models to envision the effects or causes of 
particular states or events which concern the robot 

B.7 REAL-TIME PROBLEM SOLVING AND CONTROL 

Of considerable concern is the robot’s ability to behave in real-time. Situations 
which are beyond the robot’s ability to "freeze" will be common (e.g., moving 
obstacles) and high performance will be required. The following capabilities are 
important: 

• Meta-level control. 

The ability to utilize knowledge about on-going problem solving to judge the 
relative importance, efficiency, reliability, and potential for success of alternative 
problem-solving strategies which are competing for real-time computing 
resources. The ability to make a quick and accurate judgement about where to 
focus problem-solving will be important in achieving high performance. 

B.8 INTEGRATION 

The Telerobot requires that a diverse set of software and hardware must smoothly 
integrated. Prom the point of view of the Al systems involved, two capabilities are 
critical 

• Integration of multiple knowledge based systems. 

Much of this paper describes capabilities which will be provided by knowledge 
based systems. The ability to integrate this eclectic group of systems into a 
single functional unit is critical in achieving the full "cope of robot autonomy. 
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In the past, the technical problems associated with each of the areas described 
have generally been studied in relative isolation from one another. A fully 
autonomous system, however, must rely on multiple knowledge based systems 
which can interact, cooperate, or merely tolerate one another in a common 
computing environment 

• Integration with non-ai systems. 

The knowledge based systems, part of the robot’s task planning and reasoning 
component, must be able to interact with the other non-AI systems which 
compose the bulk of the robot’s control, sensory, and human interface abilities. 
Protocols for interaction with these systems must be established. 

B.9 UNCERTAINTY MANAGEMENT 

As the System Autonomy research plan states, uncertainty management is "...the 
ability to make sensible judgements and carry out reasonable actions when world 
knowledge is imprecise or incomplete, heuristics or models have built-in 
uncertainty, or actions have uncertain effects." Any system which behaves 
intelligently and robustly in the real world must account for the inherent lack of 
precision in knowledge and ability to control the world. The following capabilities 
will be important: 

• Identification of sources of uncertainty. The robot must be able to locate and 
accommodate a wide variety of sources of uncertainty. For example, uncertainty 
can arise from partial or imprecise models, such as gaps in knowledge about the 
cause and effects of events or properties of objects (including the robot itself), 
lack of knowledge about the effects of active physical processes, or imprecision 
in modelling the intentions and plans of other agents. 

B.10 HUMAN INTERFACE AND INTERACTION 

Several human operator responsibilities must be supported by the human interface 
to the Artificial Intelligence components of the Telerobot: 

• Supervision. 

The operator must have the ability to determine, select, and specify goals for 
the robot to achieve prior to and during plan execution. The operator must have 
the ability to preview and select among alternative plans suggested by the robot. 
At all times during plan execution, the operator must have the ability to 
suspend or redirect the activities of the robot 

• Criticism. 

The operator must have the ability to evaluate and criticize plans. This could 
include the ability to make modifications to plans (e.g., manipulator trajectories; 
which are otherwise acceptable to the robot 

• Cooperation. 

The operator must have the ability to be involved in both the problem-solving 
activities of the robot as well as the actual execution of plans themselves. 
Especially in the early stage? of development the robot will often lack critical 
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knowledge about how to locate or recover from specific errors which occur 
during plan execution. The human operator must have the ability to instruct the 
robot and/or carry out the operations alone. In later Telerobot development, the 
ability for the robot and human to coordinate their activities will be important 

B.ll KNOWLEDGE ACQUISTION AND LEARNING 

There is a difficult bottleneck in the creation of knowledge based systems, Le., the 
identification, acquisition, representation, verification, and management of the 
knowledge which is required for problem solving. Knowledge-engineering has today 
developed into a skilled craft with numerous tools to aid a human developer of 
knowledge based systems. However, the volume and complexity of the knowledge 
required for the Telerobot (and other complicated systems) is sure to overwhelm 
the techniques which exist today. Ultimately, the robot should be able to perform 
much of the knowledge acquisition and maintenance problem autonomously, Le., 
learn. Machine learning has the role of resolving uncertainty, correcting knowledge 
errors or gaps, and in generating new capabilities for the robot. The following 
capabilities leading to autonomy are important: 

• Use of CAD/CAM databases. 

Effective diagnosis, assembly, inspection, and other tasks designated for the 
robot will require detailed knowledge about the structure and function of the 
objects it manipulates, such as satellites. Much of this information is expected 
to be available in computable form in CAD/CAM databases. Techniques must be 
developed for exploiting this information and transforming it into 
representations usable for knowledge based problem-solving. 

• Use of human documentation. 

Many of the tasks which the Telerobot is expected to automate have been 
designed for humans. To a certain extent, human readable documentation exists 
on these tasks or the systems to be manipulated. Techniques for utilizing this 
knowledge resource should be developed, including natural language parsing. 

• Management of massive knowledge bases. 

There are numerous problems associated with managing the volume of 
knowledge which will be required by the robot There will probably be multiple 
representations of the same information which must be consistent. At any given 
stage of problem-solving, only a subset of knowledge is required. This must be 
quickly and efficiently provided. The knowledge base management problem 
appears to subsume many traditional database management problems. 

• Learning by experience. 

When the robot makes a mistake during plan execution, and identifies the 
problem as imprecision in its knowledge, it shouldn’t make the same mistake a 
second time. In some cases, the knowledge is correct, but simply inapplicable to 
the current situation in which the error occured. In this case, the robot must be 
able to recognize in future situations when the same mistake could occur, recall 
the correction it devised previously, and implement the correction in the current 
situation. 
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• Learning by discovery. 

The robot should have the ability to fortuitously notice or bring about situations 
which are instructive. 

• Learning by teaching. 

The robot should have the ability to acquired knowledge from direct interaction 
with humans, either through factual presentation, reasoning from examples, or 
other methods of instruction. 

C. TECHNICAL AREAS FOR RESEARCH 

This section presents some of the technical areas for Artificial Intelligence 
research which should be supported by the Telerobot project over the next several 
years to achieve some of the capabilities noted above. NASA probably does not 
need to sponsor work to achieve all of these capabilities; many required 
capabilities cross application boundries and the necessary research is currently 
funded by other government agencies. NASA does need, however, to focus research 
on those areas which are critical for the near-term establishment and success of 
the Telerobot (circa 1993). The following areas are relevant: 

• Planning and scheduling. 

Near term : Research on conditional, contingency, and least commitment 
planning. Research on incorporation of physical, spatial, and temporal 
constraints and associated planning processes with task activity planning. 
Research on resource management Management of uncertainty in planning. 
Longer term: Research on multi-agent cooperation, including: Plan recognition, 
communication of plans and intentions between agents, command and 
information request communications, real-time compensation for other agent 
action discrepancies, supervisory versus distributed control issues. 

• Spatial planning and reasoning. 

Near term: Spatial model representations and databases which are useful for 
problem-solving tasks and easily integrable with knowledge based systems. 
Qualitative causal modelling of physical processes. Reasoning about spatial and 
physical constraints on task planning processes. Manipulator trajectory planning 
and collision avoidance. Longer term: 3-D robot mobility and navigation. 

• Execution Monitoring. 

.Veor term: Sensor planning issues, including: Plan action analysis to support 
sensor allocation, sensor resource allocation and real-time management, formal 
languages for specification of sensor plans, issues in active versus passive 
sensing and plan interactions, selective versus exhaustive monitoring issues, 
grain-size of monitoring, expectation generation. Monitoring issues such as: 
symbolic classification of sensor data, partial matching. Longer term: Issues in 
multi-sensor and temporal data fusion, uncertainty management, partial 
matching, noticing unexpected events. 

• Diagnosis. 

Near term: Identification of source of plan failures, including system failures, 
knowledge failures, or unexpected world states or effects of actions. Utilization 
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of multiple kinds of diagnostic knowledge, including heuristic, first principles, 
procedural Longer term : Classification of novel failures, localizing multiple 
failures, assumption changes, troubleshooting knowledge failures. 

• Error r ecov er y. 

Near term: Analysis of discrepancies on exisiting plan, plan representation and 
operators for combining plan fragments such as recovery plans. Integration with 
planning processes. Longer term: Determining discrepancy effects on planning 
knowledge (a learning problem), including: recognizing that knowledge is at 
fault, determining which know 1 edge, explaining why it is wrong, and 
implementing corrective action. 

• Simulation and prediction. 

Near term: Causal modelling for the Telerobot domain, issues in modelling 
temporal constraints and physical processes. Longer term: Dealing with bad or 
ambiguous causal models. 

• Real-time problem-solving. 

Near term: Issues in goal, planning, and execution management, including: goal 
valuation and priority, goal viability, recognition and management of dynamic 
resources required to achieve goals, interaction with human goals. Longer term: 
Issues in parallel or distributed processing of knowledge based systems. 
Real-time problem-solving software architectures. 

• Integration. 

Near term: Issues in integration of multiple knowledge based systems, including: 
Message and request communication, shared knowledge, executive versus 
distributed control. Issues in integration with non-ai systems, including 
superordinate and subordinate roles. Longer term: Integration architectures, 
including blackboards, distributed computing systems. 

• Human interface. 

Near term: Knowledge based system command languages, graphical and iconic 
display of plans, interface for supervised plan execution. Longer term: Natural 
language understanding and generation, mixed initiative dialogs, generation of 
explanations of knowledge based system behavior, communication of shared 
knowledge. 

D. METHODOLOGY 

The Telerobot project involves research and development in Artificial Intelligence 
ar.d other areas which spans the range from highly risky and innovative basic 
research through applied research and engineering to application engineering of 
operational systems. It is essential that a flow of information, techniques, skills, 
and even personnel be maintained across this span of research and development A 
recent perspective article in Science by Dr. A. M. Clogs ton of AT&T Bell 
Laboratories may be instructive: 

"...The research-development interface is a difficult enough barrier to 
surmount, even within a highly integrated R&D laboratory, and it is 
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more difficult to import research, even with the best will on both sides. 
The best way to import university research into an RAD laboratory is 
through active in-house basic and applied research groups, a channel 
that would be closed if the laboratory relied too much on external 
research.* 

- A. M. Clogs ton, "Applied Research: Key to Innovation', Science , 235, 
4784, (1987) 

Our methodological goal should be to combine a set of tightly focussed external 
research contracts with substantive in-house basic and applied research directed 
at: 


• Importing this technology and technology sponsored by other agencies into the 
Telerobot project 

• Filling in the "gaps” in basic and applied research areas which are important 
for Telerobot 

• Performing the applied research necessary to move conceptual breakthroughs 
into operational demonstrations of robot capability. 

To facilitate the importing of university technology, individual graduate students 
and professors conducting research sponsored by, or of interest to, the Telerobot 
project will be invited to visit the laboratory and work with JPL personnel for 
short periods of time. This should enhance the education of JPL personnel in 
emerging AI technology as well as provide the necessary project visibility and 
feedback to basic research efforts. 

E. CONCLUSIONS 

Thio - paper has p r es e nt e d a view of the . capabilities and areas of artificial 
intelligence research which are reqyifed for autonomous space telerobotics 
extending through the year 2000/m the coming years, JPL will be conducting 
directed research to achieve these capabilities, as well as drawing heavily on 
collaborative efforts conducted with other research laboratories. 
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1. Introduction 


Geometric Reasoning 

R.F. Woodbmy and I J. Oppenhetm 
Camegie-Mellon University 
Pittsburgh, PA 15213 



I.L Co gnfflvt robotks and space tckrobotization 

Cognitive robot systems are ones in which sensing and rcp r rjratar i on occur, from which task plans and tactics are determined. Such a 
robot system accomplishes a task after being told "what* id do, bat determines for itself "how* to do it Cognition is required when the 
work environment is uncontrolled, when contingencies are prevalent, or when task complexity is large; it is useful in any robotic 
mission. A number of distinguishing features can be associated with cognitive robotics, and one to be emphasized here is the role of 
artificial intelligence in knowledge representation and in planning. While space telerobodcs may elude some of the problems driving 
cognitive robotics, it shares many of the same demands, and it can be assumed that capabilities developed for cognitive robotics can be 
employed advantageously for telerobodcs in general 


The top level problem is task planning, and it is appropriate to introduce a hierarchical view of control Presented with certain mission 
objectives the system must generate plans (typically) at the strategic, tactical and reflexive levels. The structure by which knowledge is 
used to construct and update these plans endows the system with its cognitive attributes, and with the ability to deal with contingencies, 
changes, unknowns, and so on. Tki « japtti* nrrt rty thr topic nf M laah planning ; ir rrTffTrntm miji iuc i of representation and , 
reasoning which are absolutely fundamental to robot manipulation^ decisions based upon geometry^ r ^ 

/v c - /IX. £*' - X 

1J. Representation and Manipulation of Geometric Information • 

The uiie to this section is the essential statement of our research interest; it can be paraphrased as geometric reasoning. Consider the 
following steps in problem solving for a robotic manipulation problem: 


t . (' • 


• Identification of the environment and objects within iL 


• Identification of physical relationships between objects. 

• Generation of plans to accomplish robotic task objectives; decisions at different levels of abstraction. 

• Execution of any manipulator motion. 


Complex cognition on geometric objects is reflected in every step. The research question before us is to permit computer-based 
reasoning on geometry, utilizing constructs that human designers and users of robotic systems can employ. A particular purpose is the 
provision of an environment in which a knowledge based system (KBS) can be implemented which presents geometric knowledge 
employing the powerful concepts that humans use. and which does so uniformly with other knowledge in the KBS. 


Geometric reasoning spans issues in the representation, and intelligent manipulation of models which describe geometric objects. It is of 
general applicability over the entire range of disciplines which address real-world physical objects. While differences in the information 
requirements of various disciplines exist, the commonalities with respect to geometric information are marked. This research takes as 
us point of departure these commonalities and seeks an architecture for geometric reasoning. The methodology of the research has two 
related aspects. The first is a search for concepts around which such an architecture may be organized. The second is the development of 
a prototype sy stem, both as a medium tor exploration and as a demonstration of cocwcpt. 


The group of computer languages known as knowledge based expert system languages are of greatest interest, as it is in these languages 
ittiL much of the current work in real-world intelligent computation is being expressed. Of particular interest are object oriented 
languages, for two reasons: 1) they provide convenient means for the description of and computation on inherently hierarchical 
information. 2) techniques are rapidly developing to integrate the other main forms of expert system programming, rules and logic 
programming, into the object oriented paradigm. 
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1.3. Outline of Paper 

Section 2 discusses the Background to wort in this arcs: geometric reasoning itself (where it has been identified as such), treatment of 
constraints, certain origins within artificial intelligence p roper, geometric modelling, computational geometry, and graphics. Section 3 
contains our proposed Architecture for geometric reasoning; the r equ i rements, the concepts brought to our design, the system 
description, and a brief s ta tement of the pres e n t implementation. Section 4 contains the Conclusions and Recommendations , including 
an outline of further approaches to geometric reasoning by some colleagues at Carnegie-Meiloo University. 


2. Background 

Developments in several disciplines provide starting places and points of depa rture: 

Geometric Reasoning itself has a significant, but modest and scattered, literature. Kuipers [Ktripers 77] created a a theoretical model of 
human spatial cognition using concepts of represen tation developed by Lynch [Lynch 60]. These concepts are related together in a 
network, through which p ropa ga tion is used to perform inference. An implicit hierarchical organization is provided by using 
containment relations on spatial entities. Brooks [Brooks 81] developed a constraint based geometric reasoner as part of of the 
ACRONYM vision system. The reasoner maintains a class hierarchy (Brooks calls it a restriction graph) of geometric representations 
based upon generalized cylinders. The par ame ter s which determine a generalized cylinder are restricted through the use of algebraic 
constraints. Geometric reasoning is accomplished by algebraic and numerical methods to determine bounds on satisfiability of 
constraint sets and extremum of objective functions. McDermott described metric spatial inference, a technique to make inferences 
about the relative locations of objects based on simple descriptions. This work was extended by Davis [Davis 81]. Constraints on the 
relative locations of coordinate systems are used to define fuzzy maps which capture the constraint information in a form amenable to 
computation of queries. Davis (Davis 86] greatly expanded on this earlier work to develop a theory of cognitive mapping which 
particularly addressed issues of re pre sen tation, retrieval and assimilation. The seminal contribution of this work is the formalization of a 
means of building a map incrementally and of performing inferences on incomplete maps. AJriner [AJriner 85] built a geometric 
reasoner based on resolution theorem proving in the domain of right rectangular orthogonally oriented cuboids. Geometric objects are 
expressed as a s s er tions in Prolog and reasoning is accomplished by application of the backtracking search in Prolog using a set of rules 
about geometry. Dixon [Libardi 86] [Nielson 86] has used the concept of features to build various specialized representations for design 
problems. Features are groupings of boundary elements of a solid that provide convenient units of concept to application programs. 
Wing and Arbab [Wing 85] outlined a deductive geometric reasoning system. The main contribution of their proposal is recognition of 
the need for a clear language for geometric reasoning. 

Constraints are the basis for certain types of computation which seek to maintain some characteristics of an object constant under 
modification of the object Constraints are intimately related to dimensioning, tolenmcing and variational geometry. The literature on 
constraints is large and growing. Sutherland [Sutherland 63] created SKETCHPAD, recognized as the seminal work in the area. The 
contributions of SKETCHPAD include propagation 1 , numerical solution through relaxation and a method (pins) for linking constrained 
objects. Steele and Sussman [Sussman 80] present a language for the representation of almost hierarchical constraint networks, a 
method cf explanation of constraint calculations and a simple solution technique called local propagation . They also discuss algebraic 
transformations of constraint networks. Steele, in his dissertation (Steele 80], deeply examines the implementation of constraint systems. 
Boming (Bormng 79] developed a programming language dominated by constraints. His contributions included a strong use of the 
object oriented programming paradigm, a local and fast propagation algorithm and the use of planning for propagation. Light and 
Gossard [Light 82], demonstrate variational geometry , an instance of the relaxation algorithm used to dynamically manipulate 
parametric primitives. Gosling [Gosling 83] contributed several new techniques for constraint satisfaction. These include: the use of 
breadth-first search to plan propagation, the use of automatic transformation of constraint networks and a fast graph isomorphism 
algorithm to make frequent use of transformation feasible. 

Artificial Intelligence and related fields provide key concepts of search, hierarchical knowledge representation, inheritance, theorem 
proving and deduction. Specific results in robot task planning [Fikes 71] [Fahlman 74], geologic map interpretation [Simmons 82] and 
real- world simulation [KJahr 82] provide useful precedents. STRIPS [Fikes 71] demonstrated robot task planning in a circumscribed 
domain. In contrast, Fahlman [Fahlman 74} developed a planning system which was highly dependent on a powerful underlying model. 
The resulting simplification of the actual planner demonstrates the effectiveness of isolating geometric issues. Simmons [Simmons 
82] used a diagramming scheme separate from the rest of his program to compute adjacencies, positions and orientation of parts of 
geologic formations. Klahr [Klahr 82] discussed various approaches to geometric relationships in the development of an object-oriented 
battle simulator. His conclusions concerning the use of so-called auxiliary object strongly support an independent treatment of 
geometric information. 

Geometric Modelling grew out of early applications of computers to design and manufacture [Requicha SOj [Eastman 79]. Geometric 
Modelling iS concerned with die representation and manipulation of subsets of three ■ -dimensional Euclidean space and with die 
construction of computer systems to support these tasks. Several significant and distinct methods of representation are derived from 
Geometric Modelling. These methods are: pure primitive instancing, spatial occupancy enumeration, tree decomposition, sweep 
representation, ceil decomposition, constructive solid geometry and boundary representation. Of these, tree decomposition, constructive 
solid geometry and boundary representation are of most significance to this work. In recent years a great number of results have 
emerged in this field. 

Computational Geometry is. according to Preparata and Shames (Preparata 85] an attempt to ‘reshape - whenever necessary - the 
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classical discipline {of geometry) into its computational incarnation This field is concerned with the design and analysis of problems 
in the construction and computation of properties of geometric objects. Preparata classifies problems in computational geometry into 
five categories, each describing problems in one of the following areas: convex hull, intersection, geometric search, proximity and 
geometric optimization. From this field may be garnered good algorithms for these problems. 

Computer Graphics has spawned a wide variety of tochniqoes for the creation of images on computer output devices and for visually 
based human-machine interaction. The fundamentals of this field, especially rendering algorithms and interaction techniques have 
relevance to the present research and are of interest for the services they can provide in creating a usable geometric tooL 


3. An Architecture for Geometric Reasoning 


3.1. Requirements 

An architecture for geometric reasoning must meet several requirements. It must 

• Represent a wide DOMAIN OF GEOMETRIC ENTITIES. A geometric entity is a subset of three dimensional Euclidean 
space (R 3 ). Combinations of restrictions based upon finite described) ility, compactness and regularity can be used to 
specify a large and useful set of classes of geometric endues. A myriad of algorithms exist to represent and manipulate 
members of these classes. 

• Represent a wide DOMAIN OF SPATIAL RELATIONSHIPS. An object in space is located with reference to some 
other object or to some coordinate frame. Such a reference establishes a relation between the located object and its referent 
Relationships are a key component of modelling systems as most modelled objects are not homogeneous but consist of 
many interrelated parts. Many types of relationships exist; of particular interest here kinematic, locational and ufological 
relationships. 

• Support a wide variety of useful QUERIES on geometric data. Queries take as input a description of subsets of R 3 and 
produce as results another subset of R 3 , a vector, a scalar or some textual description. The results of queries may be 
considered as partial models of the original subset of R 3 . 

• Support the general CLASSIFICATION of geometric objects. Geometric entities may be classified by the results of any 
query on those entities. A classification scheme examines a representation of geometric information and determines if it 
denotes an object which belongs to any of the classes defined in the scheme. 

• Provide a general means to GENERATE families of objects. An incomplete model of a geometric object defines a class of 
objects. C, all of which are partially described by the incomplete model, but which differ in characteristics not described by 
the partial model 2 . A generation algorithm produces members of the class C when given as input an incomplete geometric 
modeL If the generation algorithm can produce all such members, it is said to be exhaustive. 

• Provide a wide variety of MODIFICATION TECHNIQUES on geometric data. A modification of geometric data is an 
operator which changes some aspect of the data while leaving other portions unaffected. Both object definition md object 
altering operators are modification techniques. 

• Be directly ACCESSIBLE from a knowledge based system. A geometric reasoner is intended to be used as a utility for the 
representation and manipulation cf geometric information -hin a knowledge based system. It must be accessible from 
such a system in a clear, well-defined manner. 

• Be EXTENSIBLE to include new algorithms for geometric computation, without structural change to the system. Both the 
number of applications which use geometry and the number of algorithms on geometric representations are growing 
explosively. A geometric reasoner must be able to gracefully accept the addition of new algorithms and representations if it 
is to continue to be useful. 

1 • 


3 J. Concepts 

The require menu above pose constraints on the performance of a system, not upon the means of achieving such performance. We have 
developed a system architecture within which these requirements may be achieved. This architecture is comprised of diree component 
concepts: classes of spatial sets, geometric abstractions and features. The following describes each of these concepts ndividuaily and 
'hen relates the concepts together to form the overall architecture. 

3-2.1. Classes of spatial sets 

Spatial sets may be organized into classes. A class is distinguished by certain properties that must be true for all of as members. For 
example, the class of regular poiyhedra consists of those poiyhedra which have identical vertices and dihedral angles and faces which 
are congruent, convex and regular. Classes are recursive: they may contain subclasses. A subclass is governed by all of the true 
properties of its superclasses. The class concept is familiar to many, particularly those who have studied object-one rued programming. 
It addresses three of the requirements outlined above. 


'An example li an adjacency matrix u a partial representation o t a Layout ot reo*n*lM. 


353 



1. It facilities the definition and specification of a wide domain of spatial sets. 

2. It provides a mechanism for the automatic classification of spatial seal 

3. It establishes a clear framework for extensibility. # 

A wide domain of spatial sets is naturally su pported by die class concept. Structuring geometry into classes allows the transparent 
definition of useful representations and algorithms for Afferent objects. Each class of objects, for example polygons, may have a 
specialized data str u ct u re or structures and a set of algorithms for computing such properties as displays, areas, convex hulls and 
bounding boxes. A class may partially define some its properties. Far example, the location of an object in space may be unrestricted by 
a class, or may be restricted to a subset of space. This ability to partially define prope rti es supports the meaningful creation of instances 
of a class. An ingynee displays all of the properties defined for its class, but holds specific values for properties that have been only 
partially defined by the parent class. 

The automated classification of spatial sets may be supported by algorithms which operate on members of a class ana gvaauate those 
members down the subclass tree if they meet the conditions specified by a subclass. This classification can be extremely efficient, in 
that Us complexity is proportional to only the depth of the class tree, not to the number of classes defined. 

Extensibility is addressed by the class concept as new classes may be defined by specification of additional properties on an existing 
class or set of classel All of the prope r ties of the old classes are maintained in the new class. 

3^2. Geometric Abstractions 

Geometric abstractions are representations of only a portion of the properties of a spatial set A geometric abstraction may have its own 
representation and set of algorithms for the compilation of properties represented by that abstraction. For example, the vertex-edge 
graph of a polyhedron can be simply rep r esen ted in a variety of ways and is sufficient for wireframe display. Geometric abstractions are 
organized into a hierarchy in which an abstraction is a sub-abstraction of another in the hierarchy if it can be computed from the higher 
level abstraction. Figive 3-1 shows an abstraction hierarchy for some simple abstractions on an assembly of geometric objects. 

The concept of geometric abstractions contributes to six of the eight requirements for geometric reasoning: 

1. It provides a means to represent entities within the modelling domain prior to having complete knowledge of their 
configuration. 

2. It provides a uniform representation for relationships between sets of objects. 

3. It unifies queries on objects with representations of objects. 

4. It provides a framework for the formulation of generation problems. 

5. It provides a framework for the maintenance of consistency of g e ometric information under modification. ^ 

6. It provides transparent access to appropriate levels of representation. 


Unabstracted 

Collection 


Adjacency Containment Convex Volume 

Graph Graph Hull 

, i 

i i i 

Minimum Extent Axis 

Plane °air 


Plane or 

Figure 3-1: An abstraction hierachy for collections of potyhedra 

The domain of representation discussed in Section 3. 1 is a specification of the classes of spatial sets that should be modelled by a 
geometric reasoner. It does not give guidance for creating the representations which support modelling. Geometric abstractions provide 
a means to build multiple representations, and to link those representations together into a hierarchy. Multiple representations support 
ihe incomplete definition of geometric entities as each of the representations can be considered a partial model, which captures some 
properties but leaves others undefined. Linking these multiple representations together into a hierarchy provides a framework for 
inference upon partial knowledge. Figure 3-2 demonstrates how an abstraction hierarchy supports this inference. If ail that is known 
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about a p— ***"*** geometric entity is its convex hull, then several different a bst ra ctions (minimom plane pair, plane-of, extent and axis) 
may still be compu lod. Compotation of additional infonnatioo demands that information be added to the reasoncr in rome fashion. 


Unabstracted 

Collection 

I T 1 1 1 

Adjacency Containment Convex Volume 

Graph Graph Hull 


Minimum Extent Axis 
Plane Pair 

I 

Plane of 

Figure 3-2: Use of an abstraction hierarchy far inference on incomplete information 

Information about relationships between objects may be caponed in a graph re pr esen tation which is independent of the relationship 
modelled. The only difference between different relationships is the algorithms which are used to examine the graph data structure 3 . For 
example, a graph which captures informaboo about adjacencies can be searched transitively to determine connected sub-components. A 
graph which ca ptur e s proximity information cannot be so searched as proximity is not a transitive relation. 

Geometric abstractions unify notions of queries and partial representations. A query on a geometric entity produces an abstraction or 
partial representation of that entity. This abstraction may be geometric object in its own right, or may be a graph representation or a 
vector or scalar quantity. In any case, for every query on a geometric entity there exists an entry into the abstraction hierarchy for that 
object. 

An abstraction hierarchy can be interpreted in two directions. If downward computation is defined as the direction from complete 
models to less complete models, and upward computation is the reverse, then downward computations co r res pond to queries an models 
and upwards computations correspond to generation algorithms. Far each query there can be formulated a problem in generation, which 
may or may not have a reasonable computational solution. In almost all cases the algorithm will be combinatoric in complexity. For 
example, generation of rectangular layouts from an orthogonal graph representation produces on the order of 10 6 possible layouts when 
only 10 rectangles are considered. This dual interpretation of abstraction hierarchies links together queries and generation problems. 

The abstraction hierarchy provides a means for maintenance of consistency under modifications. A change to one of the representations 
in an abstraction hierarchy may impact information contained in other re pr es e n tations. A conservative strategy is to delete all 
information dependent upon the changed information and to recompute the deleted information as it is needed. The abstraction 
hierarchy can be used to incrementally improve upon that conservative strategy by including specific modification algorithms to 
selectively update nodes in the hierarchy. For example, under isometry transformations, the convex hull of an object will be changed by 
exactly the exactly the same transformation that applies to an unabstracted collection. 

When a query is put to a model stored using an abstraction hierarchy, the hierarchy is searched for a representation from which that 
query can be answered. If the representation is not found, it is computed. If the required representation cannot be computed due to 
insufficient information being stored in the hierarchy, then that fact is returned as the response to the query. All of this can occur in a 
manner completely transparent to the application which is using the reasoner. 


3.2J. Features 

Features 4 capture the nearly hierarchical decompositions of composite objects dial humans impose upon these objects. This 
icccmpcsuion is essential to die process of understanding, manipulating and creating geometric forms, yet it has little to to -v tih 
geometry per sc. It is rather a recursive naming mechanism, that supports the aggregation of geometric entities into complex 
composites. This naming mechanism should be as flexible and general as possible, to support many different approaches to 
decomposition. Virtually the only restriction that is reasonable to place on this mechanism is that it not be self-referential, that is, that no 


’These could be called inference tig ocuhma without domg violence to common usage. 

*The ongai of the lenn “features" arises from its general usage in computer'asded design as a means of specification of portions of a modd that are of mteresi. Recently 
specific use of the term has occurred in the coo text of mrchanirai engmeennt design and machinsig [Libaidt 86) (Nielson 86). Its meaning here docs not oontrajicx these 
usages. 
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feature ext contain itself as ooe of its sub- features. The concept of features addresses four of the global system requirements: 

1. It adds the notion of an assembly of objects into the domain of representation. 

2. It provides a framework for the computation of relationships between objects. 

3. It provides a framework for the computation of queries on objects. 

4. It is a basic interface mechanism, allowing uniform access to a system. 


The fundamental domain of re pre s entation for a geometric reasoning system is subsets of R 3 . Features provide a mechanism to structure 
combinations of these sets in any way required by an application. The single limitation on features, that of noo-self-referemiality. 
imposes very little on the scheme. Features may be used to r epre sen t strictly hierarchical structures such as the instance tree of 
modelling systems [Woodbury 86] and high performance graphics packages [Brown 85] as well as structures which approach a general 
data access scheme. 


By providing a means to group together geometric entities of interest, features provide an mechanism for passing arguments to functions 
which compute relationships between entities. For example, polyhedra which represent a robot arm may be grouped together under a 
single feature and submitted to an algorithm which computes the allowable movements of the assembly from the constraints imposed by 
the joint geometries. If each of the links of the robot arm is represented by a number of polyhedra. these may be grouped into a 
sub* feature and considered as one unit for purposes of the allowable movement algorithm. The information computed by the algorithm 
is a property of the feature as a whole, not of the individual spatial sets which compose the feature. 


In a similar fashion, features may be used for the computation of queries on objects. For example, the convex hull of a feature which is 
composed of a number of spatial sets will in general be different from any of the convex bulls of the component sets. Queries on 
features compute properties which pertain to the entire assembly, not necessarily to its constituent parts. 


Features are a basic access mechanism to a geometric reasoner. They provide a means to reference geometric objects in terms of the 
semantics of the application at hand, rather than in terms of data structures which repr esen t geometric sets. A user need know li ulc more 
than the basic structure of features and the protocol of operations which apply to them to effectively build and access complex 
geometric models. 


3 J. The Concepts Together: An Architecture 

The three concepts, classes of spatial sets, abstractions and features mutually interact Out of these interactions can be developed an 
overall system architecture. 


The two concepts of classes of spatial sea and features are related by a single role: a feature may be declared as a member rS a 
particular class of spatial seL This role has cwo implications: 


I. It preserves the uniformity of features as the main access scheme -o the geometric reasoning system. The specific 
geometric models which may be part of the definition of a feature are hidden from ’-he user of the reasoner. .All diet is 
visible is the feature hierarchy. 


2. It further restricts the allowable contents of a features. A feature which is also a spatial set may contain sub-features, hit 
Jiese sub- features are constrained to be elements from the boundary of the spa cal set 5 . For example, a window regulator 5 
for an automobile may be described as having several parts, one of which is the backpiate. If die backpiate is declared :o 
belong to the class of polyhedra, di cn ail sub-feanires of the backpiaie must be a part of the boundary of the polyhedron 
which describes die backpiate. 


The interaction between features and abstractions can be expressed by a single role: Features arc die fundamental units of abstraction. 

This rule has two implications: 

1. Only features may be abstracted. An abstraction hierarchy grows whenever queries are performed upon an enuiy. Since 
features are the only enubes upon which queries are defined, they are liso the only enuues which support abstractions. 

2. Features store abstraction information at die appropriate level .An abstraction generated on a particular feature is stored in 
the absffacuon hierarchy associate with dial feature. .Any abstractions generated m the process of computation are. 
automatically stored at die appropriate level in die feature hierarchy. For example, consider Figure 5-3. Representations 
which abstract die window regulator as a whole are stored with die feature labelled wi/mow re%uiaxr. Re presentations 
which abstract only die backpiate cf die window regulator are stored with the feature labelled backziaie. 


Classes cf spatial sets and abstractions are simply related. The rule in this case is: every abstraction is represented by either a <p«oui :ci, 
3 graph, a vector or a scalar. This role has a single implication: 

l. It makes ail of die power of die cnderlying operations on spatial sets available jo die abstraction nechamsms :n xc 
eecmetnc reasoner. For example, n computing die bounding box of an assembly of objects, die reascr.er could proceed 


TSjj raincuatt jo lilowa&k contents creates » cio*e ipproiimauon :o mow TnptemenLiucnj i p»rt-*noie hieraoues. 

' \ *inoow regulator J J*e Sc vice. jssKfe i*e Jocr t in luiomootic. no?* 'Jw £t«i .1 -he *mOow j\ response jo tumui* »noo* ermn* :o ?us run* .-.e 

j*X"v*ior s'Vtfcfi ji oecincii models.) 
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Figure 3-3; Storing abstractions at the appropriate level 


from either the complete assembly or could first compute the convex hull of the assembly. In either case, by virtue of both 
representations being defined in terms of polyhedra, the same algorithm would apply. 


When the three concepts are considered simultaneously an overall system architecture emerges. Figure 3-4 diagrams this organization 
In this figure, the boxes refer to each of the concepts and the lines are inheritance relations between concepts. The top level box. labelled 
object denotes the unity of all of the concepts as data objects in an implementation. 



Sets 

Figure 3-4: The overall system architecture 


3.4. Implementation 

This implementation of this architecture a affected by the choice of programming environment- We chose object one need programming 
for this exploration as its concepts of polymorphism and inheritance map naturally onto the hierarchical structuring :i geometric 
information that is proposed here. The particular programming environment used, CommonLoops, has two special properoes. multiple 
inhen Lance and muiti- methods, dial affect the overall sysrcn architecture: 

1 . Multiple inheritance permits facile merging of die system coocepts. The class sbstractable spatial sets is forced by 
inheriting from the classes features and spatial sets and by defining a small number of methods specific to the new c ass. 

2. Multi -methods permit the specialization of methods on the basis of class membership of more than one of the method 
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arguments. This is in distinction to so-called classic message passing in which only the class of the first argument is 
significant Multi-methods simplify the creation of uniform protocols over operations which involve more than one object. 

We have implemented a simple two-dimensional geometric reasoning system based upon the three concepts, classes of spatial sets, 
geometric abstractions and features. With the lessons learned from this exercise we axe currently implementing a much larger scale 
demonstration in three dimensions, using an existing solids modelling system as the basis for geometric computation. 


4. Conclusions and Recommendations 

The architecture for geometric reasoning contained in section 3 is offered as a promising approach to representation and manipulation 
of geometric information for knowledge based systems. Its implementation is underway, and it win be evaluated for problems of robot 
manipulation. It win also be applied to problems of mechanical parts description for manufacturing or assembly. 

Additional approaches to geometric reasoning, offered by colleagues of the authors, are partly outlined here. One approach is an 
investigation of natural language function in conveying geometric information; this work has its origins both in cognitive psychology 
and in architectural research into spatial cognition [Goel 86]. Another approach, convincingly demonstrated in two-dimensional space, 
is the representation and abstraction of loosely packed a rran ge ments of rectangles in automated layout research [Flemming 86]. A third 
apprach is reflected in language design for geometric representation and manipulation [Wing 85]. A fourth approach is found in studies 
of automated assembly and disassembly of objects. A fifth approach is exemplified by the design of domain models for robot 
manipulation within complex facilities such as nuclear power plants, together with the design of blackboards for robot control a 
comparably rich environments [Keixouz 86] 
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Abstract 

This approach to dynamic scene analysis is a qualitative one. It 
computes relative depths using very general rules. The depths calcu- 
lated are qualitative in the sense that the only information obtained is 
which object is in front of which others. The motion is qualitative in 
the sense that the only required motion data is whether objects are 
moving toward or away from the camera. Reasoning, which takes 
into account the temporal character of the data and the scene, is 
qualitative. This approach to dynamic scene analysis can tolerate 
imprecise data because in dynamic scenes the data are redundant. 

Keywords: qualitative vision, dynamic scene analysis, rtlativelrur** 
facendepths- 

1 Motivation^ for qualitative vision 

For many reasons computer vision has proven a difficult task, far 
more difficult than was originally suspected. The complexity of the 
real world .3 sampled spatially and f emporaily and projected onto 
a time ordered seo/ience of frames. The description of objects, re- 
lationships and events among those objects is a signal to symbol 
transformation which requires the top-down use of knowledge (i.e. 
an interface to a/memory). Because the projection process *!oees* a 
dimension, mtetpretation must be able to tolerate ambiguous data. 
Noise compounds the ambiguity of the frame sequence. Roberts’ 
work in this 0eid, 22\ attempted to compensate for noise using 
several heuristics for line detection and a top-down modeJ-fittmg ap- 
proach. An approach to solving this problem of model-fitting under 
noise is given by Brooks 8j. In this work, he has the problem of 
unknown transforms between model and image; to solve this prob- 
lem he uses a constraint manipulation package to limit the matches 
between image and model and hypothesize other matches consistent 
with the constraints. The transforms are initially under- specified, 
then increasingly constrained. 

Two approaches have proven to be particularly restrictive. The 
drat is the to cus on single frame analysis. Eariy researchers ieit that 
it was necessary to first process one frame, and only then examine 
the subsequent frame. This is a sterile approach because it avoids 
all temporally changing scenes (i.e., things like pictures and maps), 
including most scenes of interest. The second approach which has dis- 
appointed is the careful computation of numerical features in a data 
driven manner. Examples are 3-D positions of feature points ob- 
tained via structure- from-motioo or of surface normals from optical 
Sow, shape from shading, or texture, motion parameters: v„,v v ,v ft , 
and optimization of objective functions (for citations describing these 
various approaches see: i’, 20], 24l, 27], 32 ). Most rely on an in- 
verse transformation, from two dimensions to three, and that, com- 


bined with the noise inherent in sensor and the sampling and digitiz- 
ing processes, means that algorithms providing quantitative solution 
will be inherently very sensitive to noise. 

Along with the growing interest in dynamic scenes, the realiza- 
tion that improving accuracy in a highly restricted set of features 
ao * particularly help the interpretation process some vision re- 
searchers 25] are being drawn to the qualitative approaches being 
used for common sense reasoning, naive physics and circuit analy- 
sis r 6j, [12]. The r e asons for this are that qualitative approaches 
show that it is possible to obtain useful results when solving prob- 
lems with uncertain, approximate or only signs of parameters. The 
representations of the problem domains are an attempt to capture 
the fundamental nature of the system, while avoiding the complexity 
of dynamic equations. Currently, much of the work done in qualW 
tative physics involves determining appropriate states and symbols 
and an understanding of the nature of state change. Another im- 
portant component is a simulation process, which allows one to get 
a grasp on causality. The qualitative approaches thus far attempted 
in A1 have generally included things like signs of derivatives [9) [16- 
or transitions ’10]. Computer vision is also a testbed where consid- 
erable intelligence is required. Further, the data in computer vision 
are always noisy, frequently redundant, and often misleading. 

Another approach for handling the noise for bottom- up processes, 
having support from biological vision systems, is to use a variety of 
window sizes or a collection of band- passed images. Larger sized 
operators average over a greater area, and thus ibr reasonably well 
behaved noise, the noise has less effect on the result. Unfortunately, 
the larger the window, the more likely it is computing a single result 
over two or more different pixel source populations. Some researchers 
have proposed using a set of different window sues, large ones for 
large scale and perhaps low contrast changes and smaller ones for 
more local changes 17], the scale space is a continuous vermioa of 
this t 32j. It is oot dear how to combine information among these 
many channels, partly because the channels are being used for two 
different :;ungs: detecting .o r measuring) at different scales, ana 
using larger channels to reduce nowe effects at the lower channels. 

Event detection in its most general sense locates the interface 
between qualitatively different sources of pixel population. The idea 
of event detection is * ot to smooth over noise, and thus over differ- 
ent pixel populations by using a magically chosen window size, but 
instead to detect where the pixel population changes and avoid any 
integration across that boundary. Using this paradigm are : 5j, '111. 
and [13]; also t 24j, using a finite element approach, can fracture the 
surface at appropriate places. 

This now* issue has especially frustrated dynamic scene researchers 
because it has been shown mathematically that ail 3-D information 
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(to a scale factor) U available in the optical flow field. Attempts 
to get the in format too have been fruitless because even the best ob- 
tainable flow fields are too badly corrupted. Thompson, et al., [261 
take the approach that if precise values are not computable, then 
compute the qualitative information: which segment is the occluder 
and which is the occluded. Jain [14] has also obtained this infor- 
mation for different sorts of scenes. Both use only a crude, though 
computable, approximation to optical Sow. The first uses an ap- 
proximation to the flow field called a disparity field requiring good 
feature detection and correspondence algorithms. The second uses a 
more qualitative approximation, computing the time history of pixel 
changes. But in any case it is clear that useful results are possible, 
even from the noisy data available, using and computing qualitative 
attributes rather than precise, brittle ones. 

2 Using models in computer vision 

There are a number of reasons for building or using a model in com- 
puter vision. 

1. A model provides a simplified representation. For example, the 
motion of a point may be specified with initial slate and state 
transition equations, thus it is not necessary to store, for each 
point m time, the position of a point. 

2. At some, perhaps iow. level, the data can be said to be under- 

stood when there is a model which Sts them. For example, at 
a very low level when the data St, e g., a straight line, the line 
is a model for the data, and as a line is understood, so 3 the 
underlying phenomenon giving rise to the data. This is like 
number l above At a higher level, for the model which is a 
line y - one can say e g., m is velocity and k is starting 

position At the highest level, if the model is a frame with slots, 
then the general practice is to use a prion default values for 
slots which are not filled from the data. Thus, a limited data 
set which cause a particular frame to be instantiated, triggers 
a top-down use of the model in which more is understood than 
can be derived n an immediate sense from the data. 

3. if the data can De said to fit some model, then, because in 
general, each data point need not be kept around, the inter- 
pretation processes can be made more tolerant to noise. !n 
the representation, data can be allowed to lie within a range of 
values, as signal - noise, where the noise has some understood 
or assumed statistical properties. 

4 Missing or amoiguous data can be handled by assuming the 
data exist according to the model, but are not measurable for 
wme reason This property of models is heavily reued upon by 
computer vision researeners because occlusion is rampant. 

5. When a model .s uvaiiabie. it can play the roie of a kind of 
short term memory where the integration of lata, especially 
of errorful data can oe incorporated This property was used 
:n 2 .n their ^ery early work on iynamic scene analysis, in a 
primitive fashion 

There are a a u inner of representation techniques for models. Per- 
haps the most popular is to use frames having slots IS A abeiled 
‘r vrr.e * number >f ’abejed riots which 'an re filled with 

numbers, attributes a e. symbols), variables or unxstoot&er :rames 
The procedures which manipulate these slots, for example how the 
slot with label velocity relates to dynamical equations which can pre- 
dict future positions are also part of the modelling process The 
dynamical equations are modelling the motion These sorts of cal- 
culations are not as easily represented in frames, since this sort of 
knowledge is more naturally given as procedures Interpretation in- 
volves instantiating the model from among a set of competitors that 
best match the .lata An event indicates where models, or perhaps 
uniy parameters of the models, change Confidence is a number ex* 
pressing some kind of probability that the model is correct or that 
the data measurement s correct, when that nformatioo s* available 


3 Purpose and use of chronologies 

The very earliest works in dynamic scene analysis required the rep- 
resentation of velocities and positions over time. It is not enough to 
give a simple initial state, because most motions are not descnbable 
with simple dynamic equations (consider hierarchical or non-rigid 
motions), and because they do not incorporate changes in motion 
descriptions, and because other interesting temporal characteristics 
are not included in a natural fashion. The early works did not keep 
chronologies. Instead, in ;2] for example, they kept a model of the 
scene for one time instant only, and used that to predict the model for 
the next frame. This implicitly incorporates the initial state descrip- 
tion. Robot planning frequently requires the description of several 
actions over an extended period of time. These are generally inspired 
from the approach of describing the state of the world and robot at 
each time instant (for an advanced use of this see 7'). 

Tsotsos *28] made extensive use of chronologies which were essen- 
tially time-ordered positions of points to choose among hypotheses 
for high level motion descriptions (e.g. expand, sway). His system 
chose the best hypothesis by examining the time-course of confi- 
dences of the possible schemas. This example exemplifies a major 
use of chronologies: to disambiguate local motions into more global, 
longer term motion descriptions. Other uses are to be able to predict 
future positions and circumstances, to identify interesting motions, 
and to localise events in the motions. In addition to obtaining long 
term motion descriptions, a history of events or of motions, or of 
relationships between object parts, is useful on its own, or for deriv- 
ing ether, even higher level descriptions. That is, ne may be able 
to describe oscillatory motion as such, rather than as a repeating 
sequence of position and velocity. 

Chronologies are not really models, however, because they nei- 
ther provide a simplified representation for the data, nor do they 
provide understanding. They provide a description. Chronologies 
also provide a representation in which noise tolerance, occlusion and 
integration of ^ata in a temporal fashion can be supported, especially 
under the control of temporally dependent operation. 

4 Local Temporal Inferencing 

4.1 Introduction 

When vaJues can be tied to a number one, they are quantitative. 
Permitting bounds on values, that is, restricting them to an inter- 
val on the number line, one can still do numerical operations on 
them 3 Naive physics researchers use the qualitative (symbolic) 
descriptors: increasing or decreasing. These values are obtained by 
considering the sign of derivatives, also an interval. If the sign is 
positive, the variable values are increasing We aiao use the intervals 
I-3O.0 - ), ?0-.0M. (0~ . -oc) as qualitative values. Another sort of 
qualitative value is a relative statement. For example x is faster than 
y, or a is closer than b. This sort of relation constrains the value of 
x with respect to y (and vice versa), but loes not tie the value to 
the number line. Hasse diagrams sre a graphical representation de- 
scribing such relative statements when the relation provides a partial 
ordering The qualitative examnie involving partial order is different 
from the notions of state and of symbol. It is a rompanson. The 
ordering qualitative example :s also rrore robust to noise - though 
not Decause the error tolerance j greater 

There are other qualitative relations between attributes which 
are interval in their nature, i e . which have begin and end points. 
Vilain 30 and Allen 4, have developed an interval- based tempera: 
reasoning and labelling system Their works, and those of others, are 
applicable to domains like story understanding, wn*re there tend to 
be fixed endpoints to the temporal intervals. Vere 29 has developed 
a system which will generate parallel plans for achieving g^ls within 
time constraints. 



4.3 Constraint* on domain and general description 

For the work reported here we wish to describe the relations among 
objects, without recourse to object or scene models, over extended 
frame sequences. In particular, we with the program to provide the 
relative depths among surfaces, when computable, and histones of 
surface to surface relations. 

The data for this work are tiarordered lists of occluder-occluded 
pairs and directions of motion in depth of surfaces (toward or serif 
from camera). [14) and [26) have shown methods whereby occluder- 
occluded relations may be obtained. No further data are required. 

Suppose that all objects are stationary (i.e., as in static scene 
analysis). The data provided are triples of the sort: A occludes B In 
terms of depth, x, this means, for surfaces with changes in depth that 
are negligible with respect to tnter-eurface depths, that z(A) < z(fl). 
We use the notation A < B. Occlusion data thus places a partial 
ordering on the depths of surfaces; and for static scenes, transitivity 
suffices to provide all computable depth constraints between surface 
patches. This partial ordering does not change over time. Thus, for 
example, given the data set: A < B; B < C; 3 < D, transitivity 
gives us that: A < C, A < D, and that there a a o ordering in 
depth between C and D. Inconsistencies in data and in deductions 
are trivially detected, though not trivially resolved. 

When objects are permitted to move in a plane parallel to the 
image plane the rule for combining depth constraints is again tran- 
sitivity. If there is no change m depths of objects then the relative 
depths will jot change. 

If objects are allowed to move in depth, then the depth order 
obtained by a local occlusion analysis can no longer be used as a 
sorting criterion. Transitivity dees not hold into the future when 
depths change over time. An approach to this problem is to project 
depth constraints between two objects into the future, and then use 
those derived constraints in transitive relations at the time of inter- 
est. 

4.3 Velocity rules 

There are four pftysicaily derived rules which give the projection 
into the future for the depth orderings. Motion m depth a v. If 
Stgn\v t < 0 then motion a fowsvd the observer on the temporal 

interval [t.t - At) for 5»yniv tt *^|) > 0 motion is swsy from 
the observer on the same interval Sign[v) - 3 means there is no 
significant motion in depth The four rules are: 

• rtu'el A t *• 3, and •-tt.j.iiA) - 0 and v f = 0 ==> 

■li* ji 3t~c.t 

• rule: A, • 3, and A) ‘ Oandt'd. - 0 =» 

• rui>3 At 4 - 3 t and vtt-j^n.4) • 0 and v f > J => 

t*- ii * 

• ruie4 A# «• 3* and *■- 0 anc d 

At . ,'.i * 3, * 

These ru;es are ail expressed :n 

At r 3t and < - 0 and ' - 0 ~ === > 

A* i . ^ St i - .11 

They ve referred to n ater 'ex: vs '*iucUy ^xie$ For other motions 
of A and 3. that u. for A t 3» wnete v*Ai ► 0 or vi3) < 0 
there a no relative depth information between A and 9 at time 
f - At Inconsistencies are detectable m this scheme when conflicting 
relations are derived ; cycles are uetecteuj 

4.4 Temporally local inferencing on qualitative rela- 
tions 

The data are occluder-occiudee pairs, and the direction of motion 
in lepth -toward or away from observer) From these data, one 
can derive mfront (and behind) relations for the present tune using 
transitivity of depth ordering, and for the future using the velocity 


rut*. The derivations for the future hold under the assumption that 
there is no change in the direction of depth velocity (magnitude is 
unimportant). However, we prefer not to have to deal with such an 
unstructured, open-ended future. Since the data are arriving at this 
system at each time step, we make the inferences into the future 
for one time step only. Thus, at time to, we have a set of depth 
relations *(to)- From these relations we use the velocity rules to 
derive, for the next time tj, a set of relations 4*(t|). This set of 
relations, ignoring tine and labels, will be a subset of the relations 
at to. Recall velocity rules are of the sort A < B at to plus some 
constraints on velocity of A and B =» A < B at f|. There is no 
point in applying the transitivity rule at this point, it will not add 
any new arcs. Incorporating the data at time t| will add some new 
relations. This will give rise to the set of relations *~(t t ). Now one 
applies transitivity at this point to obtain the set ♦ (*»), and the set 
of relations is ready to project into the future ooe time step again. 
See figure 1 for a layout of the order of operations on the relations. 

Thus, this system incrementally incorporates the data as it be- 
comes available. It makes no attempt to predict further into the 
future than to the next time step. A time step is defined as when 
the next datum is available. It has no memory beyond ooe time sup. 
It is focal, temporally speaking. More global temporal knowledge is 
kept elsewhere in the system - specifically, in the object histories. 

In the system there may be several relations between a given pair 
of segments. That is, each relation has two segments aad a label. For 
the segment pair, A and B, we may have, e g. 

order label 

A. < B rule 1 

* < B data 

As long as the data are consistent and correct, these inferences 
will iteratively build a consistent partial order on the segments which 
is as complete as is possible for these rules at the current time. What 
happens when a datum is incorrect? In that case we will have an 
mconxutcat set of relations. This inconsisUncy is signaled by a cycle 
m the graph. For example, suppose we have the relatioo: .*» < 3 : 
ru/el for the graph in ♦*(<). We then read the datum 3 < A. The 
graph then contains the cycle A — 3. 

Because we have labels on relations, we know what gave rise to the 
inconsistency. For the above example we know that, because there 
is a cycle, the datum 3 < A is wrong, or the ruUl applications 

was wtong cr both. Conceivably, we could trace the cause of the 
inconsistency back further into the past. For the above example, if 
the rule l application at time t - l was wrong then either the v( A) was 
wrong, t>(3) was wrong, the relation A < 3 at time t - 1 was wrong, 
or any subset of these three was wrong. For this ooe inconsistency 
involving only two objects and two relations we have axeady fingered 
as possible culprits four attributes or relations going oack only one 
time sup. Indeed, if we kept only a slightly more compel* audit trail 
the relation A < 9 at time t - t could be further tracked damn. This 
gives rise to even more possibilities of the source of inconsistency 
even more remouiy in time. 

We are not doing this for a number of reasons. The most im- 
portant of these ss than in a dynamic scene understanding system, 
one does not have the resources to spend a lot of tune and energy 
resolving past conflict; data are continually arriving, and it is better 
to have the current land future) interpretations oe correct than those 
of the past. Secondly, many culprits are fingered for each inconsis- 
tency This is a lot of over n* ad and cannot be resolved or reduced 
fiwiu 'A* .AAntaUon currently available. For a third reason, resolu- 
tion is possible oniy in the future when more data is available - there 
■a no resolution possible in the past (where the inconnsuncy arose). 
Fourth is that we rely on the fact that there are a lot of data. Even 
though some are wrong, most will be right, we do not eant to devoU 
much effort to inconsisUncy resolution because we may expect that 
future data will set things right. There is one important consequence 
of this for the impiemralauoa we do not keep sa extensive audit 
trail. We abei each arc with only the rule that most recently derived 
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So ww do not nub any attempt to undo any bad «&cta from 
pnaaiHs bod data in the past The ptnatol data an a different matter. 
Wt do, k o a twr , want the c orract data to eventually outweigh any 
inc orrec t inferences. We hare a number of option* on h tm to go 
about doing thin Essentially there are two quest io tir 

1. bow to propagate, to the next time step a r el a tion which baa a 
contradiction 

2. bow to incorporate a contradicting pair at r el a t ione into object 
histories 

We deal with this difficulty by taking the posit i on that ooe rma 
trust the current data at the current time. Any inferences from 
that data, especially into the future may be inspect, but the data 
thsmeehrea are asumed to be correct for the time sour. We could 
take the position that any relation which contradicts dele will be 
deleted immediately, and is not allowed to propagate into the future. 
But we still have the problem oi contradictory r e lat i one which do not 
have dele on either of the labels. For example, see figure 2 in which 
we show the derivation of an inconsistency. 

We are dream to the use of a certainty factor for each relation 
in order to accomodate the possibility of occasionally invalid data. 
The certainty cf dais relations will be highest. As inferences are de- 
rived, the certainty factor of those relations will decrease. There are 
a number of technical difficulties involved in dealing with certainty 
factors and getting them to be rigorously correct. We avoid these 
by relying on the fact of large amounts of mostly right data. To rig- 
orously derive a calculus of certainty factors, it is necessary to have 
a sufficiently deep understanding of the nature of the domain, espe- 
cially of the nature of the data, noise, and sometimes even s yrien 
probability values of the data, se well as assumptions of indepen- 
dence, or and knowledge of correlations la the spirit of qualitative 
processing, we wish to avoid making such restrictive assumptions 
until necessary. In this system we are attempting a qualitative ap- 
o roach in which we know there is no we, though we don’t know its 
precise precise properties. Because of the fortunate choice of using 
dynamic scenes as data, however, we can use the fact that the data 
will be mostly redundant. Thus even though rigorous derivations for 
certainties have been done 23! and may he applicable we are not 
currently investigating that direction. We just want certainty fac- 
tors to decrease with time and with transitive "distance* from dale. 
There are a number of choices on how to combine certainty factors 
when making inferences We are currently experimenting with thw. 
The problem of which of two contradictory relations to propagate 
we deal with heuristic ally : relations which are contradictory are not 
permitted to activate the transitive rule. All other relations may 
activate both transitive and velocity rules. We put this restriction 
on tranaitive-denved contradictory rules for computational reasons 
only. Many relations are derived using transitivity, and when one 
of the links w suspect, all links derived from it are suspect. Infer- 
ences whose certainties are decreasing to zero are deleted alter a fixed 
number of time steps. 

5 Chronologies 

3.1 Representational issue — indexing 

La building a chronology of depth-ordered relations among surface 
patches for use, either as a descriptive device, or as an intermediate 
data structure for further processing, there are two ways of indexing. 
The first m to organise the relations temporally In dynamic scene 
anaiysw, unlike story understanding, the data are arriving in a tune- 
srdered fashion, e g. in frame t, the is sane set ♦(•) of relalioas. at 
frame t +• l some other set ♦{• ■+• I). The chronology of relations has 
the same appearance as the data with the addition of derived rela- 
tions In this case it is easy to see what is happening at a particular 
time instant, because time is the index into list of relations, e g 


Isms rclshMI 

1 (< A 3) (< A D) 

2 (< A 3) (< A D) 

3 (< AB) (< A D) 

4 (< AB) (< BC) (< AC) 

We sss in one indexing step which r e lations exist at t=3. Given 
tbs way the velocity rules are for mulated , it is alao saner to m a ke 
predictions into the next time step. For example, if the motions in 
depth of A, B,C are negligible, then at time 4 we can make the pre- 
diction that at tiffts 5, the following refetions will hold: (< A B), (< 
BC),(<AC). 

The second indexing -'thod is to organise by relation. That is, 
ooe surface is tbs primary index, the second surface the secondary 
index. For example: 

1* 2* temporal tnlcrvel* 

A B (1, sow) 

C (4, sow) 

O (1,3) 

B C (4, sow) 

If relations persist, or are repetitious, then it saves on space to 
index by surfaces. This re pres e ntation trades off r elationsh ip storage 
for temporal storage space advantageously when r el a t i ons are long- 
lived or recur frequently. To determine at a particular time instant 
which relations are active requires inspection of a lot of data. The 
time course of relations is rawer to. access. Event marking makes de- 
riving an interval- b as e d description easy. And this method of index- 
ing is better for dealing with noise removal, occlusion and integration 
over time. 

5.3 History and world model of depths 

Despite the fact that dynamic vision has a lot of data available, 
thanks to its ramps*! redssdascy [31 j, it is both more efficient as 
well as satisfying to keep histones of relations which are indexed by 
surface. The fact remains, however, that in order to make derivations 
(or predictions) using the velocity rules and to be ai. 'e to make a 
computation ally fast statement about the relative depths at time 
sow, we keep the current List of relations, though redundant with 
histones. That is, for now we have a "temporally indexed* set of 
relations. For now as well as all the past we have object- indexed 
relations. This means that if relative depths for any past time is 
desired, though the information is calculable (indeed, was calculated, 
then discarded), from the histories, it is not immediate. Rather, the 
system will have to step through the histories, essentially re-creating 
the world for the desired time. There are certain sumlarities with 
mvuwiiMf 9j. For sow •» rmn get ad ordered relative depth map 
by doing a straight-forward topological sort [15]. 

6 Experiments 

The local temporal Lnferencmg system was implemented as described 
in a previous sect ion. We made a few adjustments, for pruning pur- 
pose, to the lnferencmg procedure as follows. 

• Relations whiefl were contradictory, e g. A < 3 and B < A, 
were not permitted to participate in any transitivity inferences. 
This is because the only result from applying transitivity on 
contradictory relations m many more contradictory relations. 
Contradictory relations are allowed to propagate into the future 
with the velocity rules. 

• Only relations derived from transitivity which had a larger or 
equal confidence factors than other relations already present 
(between the same nodes) were posted. For example, suppose 
we have the relation A < 9 with confidence TO present, then 
we derive A < 3 with confidence 35 from transitivity. That 
new relation is ignored. 
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• Data nUlioM in take* with cmMan ID (indicated mc/ = 
ID), and other nUtiov already p Meant bet wee n the same 
aodae ae the data relation were deleted. That is, if we have 
A < B becanae of a r%Ut application with c/ = D, then we 
read the datum A < B, the pr e e i ona relef edge ie deleted, only 
the daU edge remains 

We did not perform theoretically rig orou s derivations of confi- 
dence factor*. Thie ie beraaee the exact raise for combining confi- 
dences ie not important in our scheme. Confidences propagated with 
velocity rales have a •time-decay* built in. That ie, if A < £(*q);c/ : 
*» with appropriate vefodtks, then we derive A < B(ti)\cf : **, 
where x? < x. Con fi d en ce factor* are combined for transitivity rales 
by taking the min of all the relatione involved, then applying a decay 
factor (called a Spatial decay*) to the resulting number. 

We present the r e writ e of an expe rim ent in the series of figures 3 
- 7. The input ie echoed in the *in pot- list* window. Tbs * active- 
relations* window contains a list of edges with the label (reeeoo) end 
confidence. In figure 3 the data ie only the three v m otion s of the 
objects A , 3, and C. In figure 7, the data are A < C and B < A, the 
transitive closure is performed which derives B < C. Data relatione 
have confidences of 1.0. Transitive relatione, i.e., tc are decayed. In 
figure 7, the aame three relatione remain, because the velocity rale 
ridel infer them. The confidences have all decr e ased. Notice the 
confidence for the relation 5 < C, the relation originally derived 
through transitivity, ie Jess than that of the other two relatione orig- 
inally derived from data. In figure 7, we have in the data O < A\ 
note the new relation has confidence 1.0. In addition, we have de- 
rived through transitivity the new relation D < C. In thie figure 
notice that 3 < C actually has two edges. One ie a velocity rale 
edge with confidence 0.6, derived from previous B < C edge. The 
other is a transitivity edge derived from the edges you see present, 
9 < Aief : 0.8 and A < C\cf : 0.8. This did not happen in fig- 
ure 7 at time 8 b ec aus e of the nature of the spatial and temporal 
decay factors. For this experiment, the spatial decay is larger than 
the temporal decay. Figure 7 has the contradictory relation A < D 
just read in. In Sgure 7, only the maximum relation of the contra- 
dictory relations is printed out. The old A < D because of mLel is 
act drawn, though it will be propagated into the future. There are 
other ways of choosing a set of relations without cycles. For example 
one may add up the confidences on A < B relatione and on A > B 
relations, then choose the maximum of the two. 

T Conclusion, consequences and next steps 

In this paper we have described a dynamic scene analyse system 
which uses qualitative information, available with current computer 
vision abilities, to calculate relative depths between surfaces. The 
qualitative information required are motion toward or away from 
observer and occlude r-occiudee ordering. The system derives further 
relations from the data. Errors and inconsistencies are tolerated by 
requiring confidence factors to decay on each inference step. We have 
also described ua this paper our approach to r e pr esen ting histories of 
such qualitative values. 

This research has opened a number of questions. Among the 
more important is the problem of unng such qualitative calculations 
as control for other computer ▼vkmi processes, or se intial estimations 
for thoes iterative algorithms requiring them. We see this qualita- 
tive assessment ae capable cf providing a focus of attention when 
resources are limited and for making real-time dynamic scene analy- 
sis possible The integration of qualitative procedures with numencal 
ones is an interesting problem. 


Reference* 

[1] Adiv, G. "Determining three-dimensional motion and structure 
from optical Soar generated by several mowing objects,* IBEX 
7V*a* e* PutL And. ifiriau InUlL , PA MI-7, 1965, 364-401. 

[2] Aggarwal, J.K. end R.O. Doda, "Computer analysts of mov- 
ing polygonal images,* IEEE TVeae e* Computers, C-fJ, #10, 
October 1975, 966-76. 

[3J Aiefeld, G. sad J. Henberger, Introduction to intcrvol compu- 
tutionSy Academic Press: New York, 1983. 

[4) Allen, James F., "Maintaining knowledge about temporal inter- 
val^, Comm* mentions of Uu ACM , 35, #11, November 1983, 
832-643. 

[5] Bee!, Paul and Ramseh Jain, "Segmentation through symbo&c 
surface descriptions*. Proc e e din gs CVPR, 1986. 

[6| Bobrow, Daniel G. (Editor), Qualitative rtuoouinp ahost physical 
systems, MIT Press: Cambridge, MA, 1985. 

[7] Borchardt, G.C., "Event calculus,* Proccedtays; 9th Internet. 
J. Conf. on Art. Intefl., 1985, 524-27. 

[8] Brooks, R.A., "Mod d- bas ed three-dimensional interpretations 
of two-dimensional images*, IEEE TVs* section* on Putt And 
—d Mot Ism Intell, PAMI-5 #2, March 1963, 140-150. 

[9| de Kleer, J. and J.S. Brown, "A qualitative physics based oa 
confluences," in [6|. 

[10) Forbus, K., "Qualitative reasoning about space and motion,* 
in D. Gentner and A. Stevens (Eds), Venial Models, Eribauxn: 
Hillsdale, NJ, 1983. 

;11] Grimeon, W.Eric L. and Theo Pavlidis, "Discontinuity detection 
for visual surface reconstruction" Computer Vidon t Graphics, 
und Imupe Proetutnp, SO , 1985, 316-330. 

:12] Hayes, PJ., "The naive physics manifesto* in D. Mkhie (Edl, 
Expert Spotemo in the Micro- Electronic Ape, Edinburgh Univer- 
rity Press: Edinburgh, 1979. 

'13) Haynes, SM. and R. Jain "Event detection and correspondence* 
Opticnl Eoftneertnp, 35, #3, March 1966, 387-393. 

'14| Jain, Rameah, ‘Dynamic scene analysis using pixel-baaed pro- 
cesses*, Computer , August 1981, 12-18. 

[15] Knuth, D.E., Tkc Art of Computer Proprumminp, III , Addisoa- 
Wesley: Reading, MA, 1973. 

[16) Kutpert, B., "Co m m o ns e n se reasoning about causality: deriving 
behavior from structure,* in 6] 

17) Mart, David and EOen Hildreth "Theory of edge detection* , 
Pro c Rejd So c 5, 307, I960, 187-217. 

[181 Minsky, VL, A framework for repreeentating knowledge, MIT Al 
Lab Memo #306, Cambridge, MA 1974. 

19) O'Rourke, J. and N. Badler, "Model-baaed image analysis of ha- 
man motion using constraint propagation* IEEE TVaneactira 
on Putt And und Muekmc Intell, PA V/-3, *6, November 1980. 
522-536. 

'20) Prasdny, K., "Egomotion and relative depth map from optical 
flew,* Bioloficd Cybernetics, Jtf, 1980, 87-102. 

'21| Roach, J.W. and J.KAggarwai, "Computer tracking of object 
moving in space” , IEEE Ttnnouctiono on Putt And end Muckxme 
Inteil, PAMIRS #2. April 1979, 127-135. 


365 





**(i) 

** **•(«) 
*** *(«> 


(23) Shafar, G., A UatUnutUal Three y of Etiit net, Princeton Uni- 
mty Pmc PrmcaUa, NJ, 1976. 

[24) Taraopoloaa, Domain, Image analysis nmlti*rid relax- 

ation methods*, DMK Dean ea PUt Anal aad liaeUmt h uUU, 
PAUI-8, #2, March ISM, 12S-139. 

[2S| Thompson, WA, Inexact riaioa,* Precscdmja IEEE Workshop 
on Motion: Rip r ee eptitim and analysie, May ISM, 15-22. 



[20] Thompeoo, WA, KM. Match, and V_A. Benina 'Dynamic 
wlnana uujyni in optical Boar Baida* , IEEE IViandwti aa 
PM Anti tad ifachtaa hull, PAUI-7, #4, Inly ISM, 374483. 

[27) Ttai R. V. and TA. Hoang, “Uniqn en aae and ea tim a t i nn of throe- 
ilii n iwul motioa paramatara of rigid object* with coreed aur- 
Cacao,* IEEE 7Vea* aa Pet* Anal cad liaekimt InUU, PAUI-8, 
1984, 13-27. 

[28) Ttotaoa, John, *A framework for ritual motion understanding* , 
Technical Report CSRG-114, Unhranity of Tbronto, June 1980. 

[29) Vara, Stereo A., "Planning in time: window* and durationa far 
actmtiea and goala*, IEEE TVaaa oa Put Anti tad Matkint 
InltU, PAUI-5, #3, May 1983, 248-267. 

[90) VOain, Marc B., “A ayatam for raaaoning about time* , Pea- 
eeadcapa; National Conference on Artificial Intelligence, AAAI, 
Aognat 1982, 197-201. 

[31) Terry Waymoath, Panonal communication. 

[32) Within, Andrew P. •Seale^paee filtering*, Pncttiinf* 8th In- 
tamat J Conf oo Art IntaU, 1983, 1019-22. 


**(«) 


erf rmlt 

A- 4 


B 




366 









^ 37 - 

if 



A Framework for Qualitative Reasoning About Solid Objects 

E. Davis 

New York University 
New York, NY 10012 




I. Abstract 

/ Predicting the behavior of a qualitatively described 

/ svstem of solid objects requires a combination of 

geometrical, temporal, and physical reasoning. Methods 
based upon formulating and solving differential equations 
are not adequate for robust prediction, since the behavior 
of a svstem over extended time may be much simpler than 
its behavior over local time. /B us paper riisni s sc s»a first- 
order logic, in which one can state simple pnysicai 
problems and derive their solution deductively, without 
recourse to solving the differential equations, ihis logic is 
\ substantially more expressive and powerful than any 

previous Al representational system in this domain. 



2. Introduction 

To operate effectively in an uncontrolled 
environment, an autonomous robot will have to reason 
about, understand, and predict external physical events. 
In many circumstances, however, it will be necessary to 
reason about physical events on the basis of partial 
information; the objects involved may not be wholly 
perceived, or the complete physical specifications may be 
too complex to use, or the robot may need to reason 
about hypothetical or generic situations. In such cases, the 
robot will have to reason qualitatively, inferring general 
characteristics from incomplete knowledge. Human 
common sense is often verv good at speedy prediction ct 
physical events in qualitative terms: # conventional 
computational schemes are ty pically very poor at it. 

Understanding solid objects is particularly important 
in physical reasoning, and human beings are particularly 
adept at thinking about solid objects. Our objective is to 
build an AI program that can reason qualitatively about 
solid objects and" that can derive correct predictions about 
their behavior in cases where these predictions are 
intuitively obvious. This is harder than one might rirst 
guess, owing to the manv complex ways in which the 
geometry of the objects affects their behavior. 

As a first step toward building such a program, we 
have analy 2 ed the kinds of knowledge needed to support 
such reasoning, and we have defineef a formal language L 
n which this rerrr.ai 'arguage can he expressed. \Vc have 
snown that interesting problems can be solved 
qualitatively by inference from plausible axioms expressed 
in L. rhe language L is more expressive and supports 
richer inferences than in> previous representation scheme 
in this domain. We give the full details of L and its 
applications in [i]; here, we give only a sketch. 

In concentrating on the representation and 
formulation of knowledge, and postponing questions of 
algorithms or control structure, we follow Haves f2j. 
However, wc depart from Hayes' research program in 
some respects. We do not attempt to model "naive* 
physics; rather, we have made free use of Newtonian 


mechanics, including concepts that have no com monsense 
analogue, such as total mechanical energy. Also, our 
proofs are lengthy, violating Hayes' dictum that obvious 
facts should have short proofs. 

The mathematics used here is not " qualitative" in the 
restricted sense of representing quantities purely in terms 
of order relationships and constants 13]. Such a 
representation is too weak to support tne inferences 
needed in this domain. 

We have chosen two kinds of problems as foci for 
our analysis; predicting what happens when a die is 
dropped inside a funnel (Figure 1) and what happens 
when a block is dropped onto a table (Figure 2). 

|| 
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Figure 2: A block ;s released onto a tacle. 

Different forms of these problems involve a rich, 
interconnected bodv of geometric and physical knowledge 
for their solution. This paper will focus primarily on the 
"die in the funnel" example. 

3. Background 

Several previous AI projects have studied the 
qualitative phvxics of solid objects. For example. 
FahJman's BUILD program [4) determined the stability of 
a tower of poiynedrai blocks. Dc KJecr's NEWTCN [5] 
predicted the behavior of a point mass sliding on a 
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constrain;. Bundy’s MECHO [6] used force analysis and 
conservation laws to make physical predictions in 
situations of specialized format. Forbus’ FROB [7] 
predicted the behavior of a point mass flying among 
constraints. Funt’s WHISPER ]8] predicted the oebavior 
of a collection of objects by simulating it in a retina-like 
image. Novak’s ISAAC [9] identified English-language 
programs of fixed form, and applied special case 
equations to them. Shoham [10J analyzed the local 
mobility of an object within constraints. 

All these p r og r ams provided valuable insights. They 
were, however, limited m geometrical expressivity and in 
the range of physics understood. Of these systems, oniv 
BUILD dealt with three dimensions; and only MECHO 
dealt with the motion of extended objects. Only a few 
kinds of physical interactions were considered. 

Another limitation of these programs was more 
subtle, but more fundamental; thev were based almost 
entirely on extrapolating differential behavior. To make a 
prediction, the program first determined how each state 
of the system will tend to change, and then extrapolated 
these changes to predict a continual trend of change up to 
the point that the structure of the system changes. This 
extrapolation could be done qualitatively, as in FROB and 
NEWTON, or symbolically, as in MECHO, or using 
poim-by-point simulation, as in WHISPER, or by 
numerical integration, as proposed by McDermott and 
Bemecky (personal communication). 

For example, FROB [71 predicts the behavior of a 
bouncing ball m a well bv dividing physical space into 4 
regions (the interior of the well, the bottom, and the two 
sides), and dividing the velocity space of the ball into nine 
(motionless, up, down, left, right, and the four 
quadrants.) (Figure 3) There are thus 27 possible states of 
the system. (4x9-9 impossible states). The laws of 
physics are then used to determine which transitions 
between states are allowed, and thus a transition graph of 
states is developed. FROB predicts that the system 
follows a path in this transition graph, ending in a stable 
state of rest. 


There are two further arguments. First, the sequence 
of states traversed depends delicately on the exact shapes, 
sizes, and physical properties of the die and the funnel, 
while the conclusion that the die comes out the bottom is 
robust with re sp ect to small variations in these 
parameters. Therefore, if the problem is specified with 
some small degree of imprecision, simulation will either 
be impossible, or involve some monstrously brandling 
tree of possibilities. But in qualitative reasemng the 
prediction that the die comes out the bottom should be 
almost as easy with imprecise data as with precise. 
Second, the complexity of simulation goes up rapidly with 
the number of interacting objects. In figure 4, for 
example, with one die inside another dropped inside a 
funnel, the set of system states is the cross product of the 
possible interactions of the two dice with the possible 
interactions of the outer die and the funnel. Nonetheless, 
the prediction that the two dice wiM come out the bottom 
is intuitively almost as easy as with only one die. 
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Figure 3: Discretized space and velocity in FROB 


However extrapolation is dene, simulation is 
inadequate for robust prediction. In this kind of analysis, 
each different set of boundary conditions is a different 
system state Each such uate must be separately ictcctcd. 
categorized, and analyzed, and the systems progress 
through these states must be recorded. Often, however, 
such a categorization is difficult and pointless. Consider 
the problem in figure l; a small die is released inside a 
large steep funnel. Many states are possible: the die may 
be m free-fall; it may be colliding or in continuous contact 
with the top or bottom pan of the funnel funnel on any of 
eight vemccs. twelve sides, or six faces; it mav be 
spinning, sliding, or rolling, up. down, or around the 
funnel. But the prediction that the die comes out the 
funnel does not require the enumeration of the states and 
the paths through them. 


Figure 4: One die inside another released inside a funnel 


In short, formulating and solving differential 
equations is an inadequate technique in this domain, since 
the behavior of these physical systems over exiended time 
is often easier to characterize * than their behavior over 
local time. A powerful physical reasoning system must be 
able to infer the general quality of a course of events 
from broad characterizations of the physical properties of 
the objects involved, without calculating each subevent. 

The programs cited do use some techniques besides 
simulation. MECHO and NEWTON use energy 
conservation to prune possible system behaviors. .Any 
state with more mechanical energy than the starting state 
can be ruled out as a possibility in all future states; tor 
example, the die cannot come but the top of the funnel. 
FRO& predicts that the system ends :n a stable state. We 
believe that effective qualitative reasoning requires more 
inferences like these, and less use of simulation. 

A natural knowledge engineering approach would use 
rules that state the desired prediction, such as A >mail 
object released inside a steep, laree-mouthcd funnel will 
fall out the bottom." But rules of this kind arc inadequate, 
and have rightly been rejected by previous researchers. 
Anv single such rule covers only a small class of 
problems; covering large classes of problems requires 
many separate disconnected rules. In particular, a rule like 
the one suggested above applies only when the die and the 
funnel arc uie only objects involved. .As soon as another 
object enters, the rule gives no guidance. Thai is. such 
rules are not compositional across objects Even without 
other objects, if we allow wide ranees in the shape of the 
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die and the funnel, the conclusion” will sometimes apply 
and sometimes not. Since there is no simple, general rule 
for when the die comes out the bottom, a different rule 
must be stated for each special geometric case. 

Maintaining a knowledge base with many special case 
rules is not effective. First, the knowledge base will have 
to be large and inefficient. Second, if a new case is not 
precisely covered in pre-canned categories, the system 
cannot even begin to deal with it. Third, this approach is 
aesthetically distasteful. A well-designed system should 
use similarities amone different cases of a die falling 
through a funnel, ana similarities between this problem 
and similar problems, such as a die shot through a tube, 
or a die dropped into a box. The analyses of these cases 
ought to have more in common than the use of rules 
which are syntactically similar. Finally, it seems plausible 
that an integrated system of rules will support learning 
better than a tabulation of special cases. 

4. Examples and Analysis 

We propose that the "die in the funnel" can be 
analyzed as follows: (i) Due to the topology of the funnel, 
if the die goes from inside it to outside it. the die must 
either exit the top or exit the bottom, (ii) Since the die is 
dropped from rest inside the funnel, it cannot have the 
energy to e.v* le top of the funne. (iii) There is no 
stable resting point for the die inside the funnel, since it is 
smaller than the funnel's mouth, and the funnel’s sides are 
steep, Tv) The die cannot stay forever moving within the 
funnel, for its kinetic energy will eventually be dissipated. 
Therefore, the die must exit the bottom of the funnel. We 
claim that in most cases where common sense predicts 
that the die will come out the bottom, it will be possible 
ro carry out such an analysis, and to support the substeps 
by inferences from general rales. Different problems will 
vary in the justifications of the substeps. 

Related problems wul share parts of the analysis. For 
instance, in predicting that a die in a small-necked funnel 
will come to rest at the top of the neck, we may use the 
identical arguments (i) tnat the die must either exit the 
top. exit the bottom, or stay inside; (ii) that it cannot exit 
the top; and (iv) that It cannot stav inside in a perpetual 
state of motion. The araument inn that it cannot rest 
stably inside the funnel must be modified to an argument 
that :t can only rest staoly at the top of the necx of :ne 
funnel; and the additional argument must be made that it 
cannot exit the bottom of the funnel, since the orifice is 
too small. 

This analysis avoids both problems discussed in 
section 2. We can avoid analysing, or even Jetermjnmg, 
the states of motion of the die inside the funnel; ail **e 
need to derenninc is that the die cannot rest stablv inside. 
Different categories of problems are analysed ri similar 
but not identical ways from general principles. 

In the rest or* this section, we look at variations of 
this example, and show how this analysis can be applied. 

We begin with a sirreie case t figure 5). The die is a 
uniform sphere. Hie funnel is the surface of revolution 
aoout a vertical axis of a pianar figure with a convex inner 
side. Fhe radius of the die is less than the radius of 
revolution of the funnel. The steps of the argument arc 
easily Tiled n. The v:o and bottom of ‘he Tnnei ire 
the oniy onflces of free soace connecting the inside of me 
funnel with its outside. Therefore, if the die is to go from 
mside to outside, it must go through the top or the 
bottom, (ii) Since the die is spherical. Its center of mass is 
m its interior. Since the top of the funnei is horizontal, 
and directed upward, if the die were to exit it. each point 
n the interior of the die would be above the top of the 
funnel at some time. In particular, the center of mass 
would be above the top at some time. But the die started 
out from rest beiow the top of the funnei, and there is no 
source of additional energy for the die. Therefore, the die 
cannot come out the roc i iii) By a geometrical argument. 


the die can only abut the inside of the funnel in a single 
point. A uniform sphere can be stably supported at a 
single point oniy if the supporting surface is horizontal 
there. The inner surface of the funnel is nowhere 
horizontal. Hence there is no resting place for the die 
inside the funnel. 



. Figure 5: A spherical die inside a radially symmetric funnel 


4.1. Out the top, out the bottom, or stay inside 

We now consider how this argument cm be 
generalized and modified. (Further modifications are 
discussed in [1].) Part (i), that the die must cither exit the 
top, exit the bottom, or stay insid does not require that 
the funnei be a solid of revolution; it requires oniv that 
the funnel be a tube with oniy two orifices. W* can 
weaken the condition further, and require oniy that ail 
orifices other than the top or the bottom be too small to 
let the die through. Determining whether i die can go 
through a hole is an easy geometric calculation for various 
special cases. 

4.2. Not out the top 

Part (ii), the argument from energy conservation that 
the die cannot come out the top, depends on the die t>eing 
convex and on the center of mass of the die stoning out 
below any pan of the top. Convexity is mlv _sed to 
establish that the center of mass of the object .s h its 
interior. If this can be done otherwise — for examine, by 
exact calculation, or by establishing that the object snape 
is a small perturbation of a convex shape. — mat is 
sufficient. 

A still weaker sufficient condition is that the -:n%ed 
filling in of the die is convex. The ringed rilling n of a 
'three-dimensional shaoe 5 is defined os follows: Consider 
iny pianar cross section of S . Let C be any simple dosed 
curve that lies entirely in this cross section. Let ; x a 
ooint in the riane in "me inside of C. Then p .% n the 
ringed filling in of 5. Figure o) 

Let 5 be the shaoe of some o*. ect O, ai.d let * :e the 
ringed Tiling in of a. .Assume R is convex. Geanv. R is 
equal :o the convex r.ud of S . R contains .r.e renter ar 
mass of O. Let C be a dosed curve lying :n 5 and in a 
plane containing the center of mass of 0 If C goes 
through a planar surface, then so does every point nside 
C. rhus, if O exits the top of the funnei. then the .enter 
of mass of O must likewise, and the proof goes through, 
lluis we can establish step < ii) r 'or such shapes is a torus, 
a wiffle bail, or a cratered convex shape. 

4 3 . No resting point inside 

Part (iii). the argument that the die cannot rest aside 
the runnel, depended m our first example an the strong 
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Toms T is cut by plane S 



Cross section of T bv S 
C is a curve in the cross section 
p is a point inside C. 

Figure 6 


assertions that the die was a uniform sphere and that it 
could contact the funnel only in a single point. We can 
easily generalize to nearly uniform, nearly spherical dice. 
The following formula holds: !et 0 be the siope of the 
support; let jx be the coefficient of friction; let be the 
maximum angle between the line from the center of mass 
to a point on the surface and the normal to the surface at 
that point (Figure 7). The ball can stand still only if 
p,^tan(9) and o>0. Similarly, if one die is a spherical 
shell containing another die, they rest stablv only if the 
joint center or mass of the two dice is located directly 
above the contact point of the outer die with its support, 
and the inner die rests stably inside the outer die. 



N is the normal to the sun ace. 
c is the center of mass. 

I is a line through c. 

3 is the angle between N and 1. 

Figure Distorted -chore 


If the die can contact the funnel in several points with 
different surface normals, the analysis becomes harder. 
The wider the range of the horizontal component of the 
surface normals at contact points, the steeper the slope 
must be, for the normal forces at the various contact 
points will tend to act against each other, and thus 
generate larger friction forces. The following rule holds: 
Let A be in contact with 3 . Let 9 be the minimum slope 
of the surface of B at a contact point. Consider the 
horizontal components of the surface normals of 3 at the 


contact points, and assume that there is some direcboo 
which lies within some small angle 6 of all these 
horizontal components. Let the coefficient of friction be 
fi. If *x<cos<t>tan0, then A will slide downfl. 

Combining ail the different ways in which the resets 
(i), (ii), and (lii) may be established, and all the wan in 
which their geometrical preconditions may be satisfied, 
gives a rich, interconnected body of results, all with the 
conclusion, “The die falls out the bottom of the funneL* 


5. The Block on the Table 

The behavior of the block on the table can be 
analyzed using a similar argument. After the block is 
released, it will fall to the table, tipple over a bit, and 
then move akmg the table in some combination of sliang, 
bouncing, and rolling. (Figure 8). It can be estimated bow 
long it will take for the faction involved in sliding and the 
inelasticity involved in bouncing to consume all the energy 
gained in the fail and the tipple, and how far the block 
can travel during that time. A similar estimation can be 
made for roiling, as lone as the object rolls suffiaeatly 
poorly. If the surface of the table is uniform, and if Qese 
motions will not bring the block off the edge of the table, 
then it can be predicted that the block will attain a stable 
state of rest within the estimated time, and within the 
estimated distance of the point of release. 



Block released. Block hits table. 



Block tipples over. Block bounces 

and slides to rest. 


Figure 8: A block settling or a table 


6. The Underlying Knowledge 

6.1. Geometry 

The arguments in section 3 used several different 

kinds of geo”metric knowledge, irviuding: 

• The ability to name and describe particular point sets 
that are connected to objects and are useful for 
physical reasoning, such as the too and bottom n a 
runnel. These are 'called "pseudo-objects" in cur 
system; the problem of constructing them effecuvciy 
is the same as the problem of constructing metric 
diagrams in FROB r 3l. 

• Topological predicates. For example, the runnel 
forms topologically a box with two orifices, ana die 
die starts outlnside the box. [11] 

• The use of a property Quantified over ^oil 
irregularities of a certain kind in on object. For 
instance, the funnel has no holes large enough to let 
the die through other than the top ana bottom. 

• Special shapes, such as spheres and surfaces of 
rotation. 

• Inequalities on metric dimensions. For example, the 
radius of the die is less than the radius of the fuxari. , 
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• The bounding of the ranee of the surface normal over 
a part of an object’s surface. For example, we wish to 
say that the slope of the funnel is everywhere positive 
in its inner surface. 

• Convexity and related properties. 

Any adequate geometric language will be strong 
enough that these, or most of these, can be either 
expressed directly or inferred. 

6.2. Temporal Logic 

Our temporal logic follows McDermott’s [12]. A 
scene is an instantaneous snapshot of the universe. In our 
domain, a scene specifics the positions and velocities of all 
objects. A chronicle is a function from the time line to 
scenes. Chronicles include aJI continuous motion of 
objects through space, not just those that are physically 
possible. 

The velocity of an object at an instant is defined to be 
the limit of its velocity from preceding time. Thus, we can 
speak of the velocity of an object at the instant of a 
collision. 

The ’frame’ or " persistence" problem of determining 
what remains true over time [l_. 13] docs not arise. 
There are two classes of predicates in the domain. The 
first class includes predicates that depend on position and 
velocity of objects, rhese are not .assumed to remain 
constant over anv interval unless proven to Jo so. The 
second class includes structural predicates, depending only 
on the shapes and material properties of the objects. 
These are alwavs constant over the problem, and so are 
defined atemporailv. (The closed world assumption is 
made explicit through the predicate ’isolated! 00,C)'\ 
which asserts that, during C. no mobile object in the set 
of objects (X> ever comes into contact with any object 
outside 00 . ) 


6.3. Physics 

The world consists of a finite >ct of solid objects 
moving in space through time. Objects are rigid 'and 
indestructible. The interior of objects may not overlap. 
Objects have two internal properties besides their shape: a 
distribution of mass, ind a coefficient of elasticity, which 
determines how the object behaves in a collision. .Any pair 
of objects have a coefficient of friction, which determines 
the frictive forces between the objects. 

Objects are subject to four kinds of forces: a uniform 
downward gravitational force; normal forces, which act to 
prevent objects from overlapping; friction; and a weak 
drag force, which dissipates kinetic energy. Certain 
objects siq fixed: they do not move, whatever” the forces. 

Necessary physical deductions include the following: 

• Determining whether a set of objects can attain a 
stable scene while certain geometric conditions hold. 

i Finding constraints on the location of the center of 
mass or an object or a set of objects. 

• Resolving a set >f forces, and determining motion 
under tnose rorces. 

• Predicting that the existing structure of contacts 
between objects will chance.” 

• Predicting a collision. 

• Predicting the result of a collision. 

• Determining whether a chronicle violates a 
conservation law. 

• Characterizing the paths that an object can take 
without coming to werlap other objects! 


7. Ontology 

The ontology for our language requires a number of 
sorts of individuals. 

Quantities. Instants of time, quantities of mass, 
quantities of energy. These are modelled as real numbers. 

Points and vectors. These are modelled as elements of 

R\ 

Point sets. Subsets of R 3 . 

Vector fields. These are functions from some point 
sets to the space of vectors. For example, the surface 
normals to an object in a fixed position, directed outward, 
form a vector field. 

Rift id mappings. Mappings from to /? 3 which 
preserve distance and handedness. ITicsc specify a change 
in position. 

General velocities. The derivative of a rigid mapping. 
A general velocity is the composition of a linear velocity 
and an angular velocity about a specified axis. 

Objects . These are primitive entities. The shape of on 
object is the point set tnat it occupies in some particular 
standard position. This is assumed to be connected, 
dosed, and normal. 

Scenes. A scene is a snapshot of the world. Formally, 
it is a function which maps an object to a pair of a rigid 
mapping, giving the position of the object, and a general 
velodty. The place of an object in a scene is image of the 
object shape under the mapping associated with the object 
in the scene. 

Pseudo-objects. These are point sets that "move 
around" with objects, like the hole of a doughnut, the 
opening of a bottle, or the center of mass of anv object. 
Formanv, a pseudo-object is a pair of a source object and 
a point" set, designating the point set occupied by the 
pseudo-object when the object is in standard position. 'Fhe 
place of a pseudo-object in a scene is ne image of ts 
shape under the ..tapping associated with its source object 
in the scene. 

Chronicles. A chronicle is a function from an interval 
of time to scenes. 

All chronicles arc subject to the following constraints: 

i. .All scenes in the range of the chromcie have the sane 

objects in their domain. 

ii. Objects move continuously in space. 

iii. Object velocities arc continuous from previous time*. 

iv. The velocity of an object is the derivative of -ts 

position. 

Chronicles do not have to be phvsicaily possible. We 
use the predicate "phys-poss(C)" to distinguish chronicles 
tnat obey the laws of phvsics. 

8. Axioms for Physical Reasoning 

Based on the above ontology, we have developed a 
first-order language L and a set* of axioms adequate *o 
solve the first '"’die in the runnel’’ example. Vnc complete 
inaivsis is rather lengthy: the language uses about mnery 
non-Ioeical terms, not including ;he”star.card arithmetic 
venerators, and the anal v sis involves about 140 axioms 
Most lover two thirds) n the terms and axioms arc 
purciv geometrical; the rest relate to notion and z 
physics." We give below three sample axioms, and trie 
complete statement of the ’die in the funner example, as 
illustrations. 

Geometric .Axiom: Smoothness and the value of the 
surface normal are local properties or the boundar. 
Specifically, f two bodies share part of their boundar.. 
then, at any interior point of the overlap, one is smocm 
iff the other is smooth, and their surface normals arc 
either parallel or anti-parajleJ. 



[ body(XU) • body(XX2) r 
XXA Q boundary (XX 1) n boundary (XX2) .\ 

X 6 interior! XXA) \ smooth(XXl,X) ] > 

[ smooth(XX2 JC) \ 

[ surf-norm(XXl,X) = surf-norm(XX2 JO v 
surf-norm(XYlJY) = -surf-norm(XX2.X) ] ] 

Axiom of Motion: If an object O has zero velocity in 
every scene of a chronicle C. then it stays in the same 
place throughout C. 

f Scscenes(C) > vclocity(0*S) = 0 ] > 
l Vji sz Sl€scencs(C) S2€scencs(C) > 
mapping(Sl,G) = mapping(S2,0) ] ] 

Physics axiom: The energy' of an isolated set of 
objects OO never increases in a physically possible 
chronicle C. 

[ phys-poss(C) ' isolatcd(0O,C) * T\<T2 ] > 
energy (GO, scene(C,n)) - energy ( CO, scene(C,T2)) 

Problem statement: Consider a spherical die, and a 
radially symmetric funnel. Assume that the inner radius 
of the funnel is greater than the radius of the die; and that 
the inner side of a radial cross section of the funnel is 
convex. If the die is released inside the funnel, and the 
funnel is held fixed far from the ground, then the die will 
eventually fall out the bottom of the funnel. 

Constants of the example: 

odie — the die 
ofunnel — the funnel 
c — the chronicle 

xx-pfunnel — the planar form from which 
the funnel is generated 
xx-centcr-line — the axis of the funnel 
xcenter — a point on the axis of the funnel 

Assumptions: 

sphere! shape! odie)). 

( iTic die is a sphere.) 

mobilef odie). 

(The die is net fixed.) 

shape( ofunnel) = 

sol id-of- revolution (xx- pf unnei. xx-centcr-Iinc). 

1 The funnel is the solid of revolution of xx-pfunnel 
around xx -center-line. ) 

p'anari xx-pfunnel I j xx-ccnter-line). 

(xx-pfunnel is a radial cross section of the funnel.) 

convex-side! inner-dside(xx-pf unnei. xx-ccntcr!inc). 
xx-pfunnel). 

(The inner boundary of x x-pf unnei with respect to 
xx -center- line is convex.) 

distance! xx-pfunnel .xx-ccn ter line! > radiusiodic) > 0. 

(The radius of the funnel is greater than the radius of the 
die.) 

xx-.vnteriine - -nake* : inei xcenter. vup‘. 

(The axis or the runnel is vertical.) 

si andar d- positio n( of unnei. start scenefc)). 

(The funnel is onented in standard position.) 

fixed! ofunnel). 

(The funnel is fixed.) 

isolated! {odie. ofunnel. oground}.c). 

(The die is isolated from every thing but the funnel and 
the ground.*) 


XF €shape(ofunnel) * XG€ shape! oground) > 
height(Xf) - hcignt(XG) > diameter! odie) 

(The funnel is more than the diameter of the die above 
the ground.) 

infinite(c). 

(The chronicle is eternal.) 
phvs-poss(c). 

(The chronicle is physically possible.) 

motionless(odie,startscenc(c)). 

(The die starts from rest.) 

place! odie, startsccnc(c)) C 

tube-inside! shapc( ofunnel). s- tube- top! shape! ofunnel)), 
s-tube-bot(shape(ofunnd))) 

(The die starts from inside the funnel.) 

Prove: 
cxits( odie, 

pseudo-objea(ofunnel,s-tube-bot(ofunnel,vup)), 

c). 

(The die exits the bottom of the funnel > 

9. Conclusions 

The strengths and limitations of this theory are 
evident. On tnc positive side: Using pure first-border 
logic, we give a formal analysis of a~class of problems 
beyond the" scope of any previous AI theorv. Our analysis 
suggests that a qualitative physics for solid - objects shouid 
include the following features', among others: 

• A rich geometrical theory, including topological, 
metric, and differential descriptors.'' and special 
shapes. 

• An account of the behavior of physical systems over 
extended intervals of time. Such an account shouid 
incorporate constraints placed by one object on 
another; conservation laws, especially conserv ation of 
energy; the principle that a physical system tends 
towards a stable resting point; and an account of the 
net effects of collisions 'over extended time periods. 

• The ability to determine the existence of a stable 
configuration of objects within qualitatively dcscntal 
geometrical constraints. 

• The ability to calculate, exactly or qualitatively, 

important physical parameters such as the center of 
mass. [14J • 

• The ability to bound the effect of small perturbations. 

On the negative side: We have not shown that ihis 
type of analysis is extensible to cover ail. or most, 
qualitative reasoning in this domain. We have not shown 
mat such an extension would be. in (he long run. my 
more parsimonious than simply enumerating special cases, 
is in the rule-based method rejected in section 2. We have 
not shown that any effective computational methods can 
be developed on the basin of this theory. We cannot p\e 
a final resolution to these problems until we have 
implemented a working system, and determined the ranee 
of problems that it is adequate to address. 

We plan to begin implementation X ievclopinz an 
adequate geometric representation and inference system. 
Ultimately, we want to implement a physical rex^mng 
system with ail the features mentioned above. 


W> need :hc ground, because nhervkjse -tie hypotheses ire 
nconsistcnt »trh the moms fhe attorns j>scrr that an r.i nj-e 
c.nrortictc must come to an end n a Nteadv 'Ute S.nce wc show hut 
:nere s nor ^rcjdv sfate tor *he n Me funnel. *c must pro' ae : 
* rh the ground to rest on 
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